# Intrinsic-Rank Filter Memory for Actor-Critic

- ID: 2142
- Canonical URL: https://synthcore.org/idea/2142/intrinsic-rank-filter-memory-for-actor-critic
- API JSON: https://synthcore.org/api/idea/2142.json
- API Markdown: https://synthcore.org/api/idea/2142.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.11750](https://arxiv.org/abs/2608.11750)
- Category: architecture
- Solves: stability, sample-efficiency, scalability
- ML areas: rl, rnn, world-model, training-dynamics
- Math tags: control-theory, dynamical-systems, linear-algebra, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace an oversized recurrent hidden state or raw history stack with a causal filtered input-output lift followed by an SVD-selected bottleneck. The actor, critic, and Bellman regression operate only on the identifiable memory coordinates, preventing deterministic null directions from being fitted as if they were independent state variables.

## Mathematical statement

For the continuous-time plant \(\dot{x}=Ax+Bu,\ y=Cx\), construct a measurable filtered-history vector \(z(t)\in\mathbb{R}^{d}\) from causal stable filters driven by \(u\) and \(y\). Although \(d\) may be large, the paper shows that the intrinsic dimension is \(r_{\mathrm{int}}=2n\) for SIMO systems and \(r_{\mathrm{int}}=n(m+1)\) for general MIMO systems, where \(n\) is the hidden state dimension and \(m\) is the input dimension. Given a data matrix \(Z=[z(t_1),\ldots,z(t_N)]\), compute \(Z=U\Sigma V^\top\) and retain \(U_r\), the left singular vectors with \(\sigma_i/\sigma_1>\tau\). The neural memory is \(h_r(t)=U_r^\top z(t)\). A linear-quadratic critic uses \(V_\theta(h_r)=h_r^\top P_\theta h_r\), while a neural critic or actor receives \(h_r\) instead of \(z\). The useful property is removal of deterministic null directions: the Bellman feature matrix has numerical rank approximately \(r_{\mathrm{int}}\), rather than the ambient filter dimension \(d\).

## Key formulas

- $$r_{\mathrm{int}}=\begin{cases}2n,&m=1\\n(m+1),&\text{general MIMO},\end{cases}$$
- $$Z=[z(t_1),\ldots,z(t_N)]=U\Sigma V^{\top},\qquad h_r(t)=U_r^{\top}z(t),\quad r=\max\{i:\sigma_i/\sigma_1>\tau\},$$
- $$u(t)=\pi_\theta(h_r(t)),\qquad V_\phi(h_r)=h_r^{\top}P_\phi h_r,$$
- $$\kappa(\Phi_r)=\frac{\sigma_{\max}(\Phi_r)}{\sigma_{\min}(\Phi_r)},\qquad \Phi_r=\text{Bellman-regression features constructed from }h_r.$$

## Implementation notes

1. Integration point: use a fixed bank of stable first- or second-order continuous-time filters driven by the measured output \(y_t\), action \(u_t\), and optionally their derivatives estimated by finite differences. Discretize each filter, concatenate its states into \(z_t\), and insert the bottleneck \(h_t=U_r^\top z_t\) immediately before the actor and critic. For a recurrent neural policy, feed \(h_t\) to a small GRU rather than feeding the full filter vector. 2. Pseudocode: collect an exploration rollout; update each filter with \(z_{t+1}=A_fz_t+B_f[y_t;u_t]\); form \(Z\); compute its SVD; choose \(r\) using \(\sigma_i/\sigma_1>\tau\), with \(\tau=10^{-3}\) initially; set \(h_t=U_r^\top z_t\); evaluate \(a_t=\pi_\theta(h_t)\); fit the critic from TD targets \(q_t+\gamma V_\phi(h_{t+1})\); update the actor with the critic gradient. Recompute \(U_r\) only between rollout epochs and align new singular vectors by Procrustes rotation to avoid artificial jumps. 3. Computed from the paper: the target intrinsic-rank formulas \(2n\) for SIMO and \(n(m+1)\) for MIMO, and the use of reduced filtered coordinates in Bellman iteration. Estimated empirically: the SVD spectrum, numerical rank, Bellman feature matrix \(\Phi_r\), and condition number. 4. First cheap experiment: use a two-state continuous-time MIMO plant with \(n=2,m=2\), so the predicted intrinsic rank is \(r_{\mathrm{int}}=6\), while deliberately constructing a 20- to 40-dimensional filter lift. Compare SAC or deterministic policy gradient with the full lift, the SVD bottleneck at \(r=6\), and an incorrectly compressed \(r<6\) model. Measure TD residual variance, \(\kappa(\Phi_r)\), critic loss resets, and closed-loop regulation cost. The quantitative prediction is that singular values after the sixth remain near numerical noise, \(\operatorname{rank}(Z)\approx6\), and the reduced model should have at least an order-of-magnitude smaller condition number than the full regression. Performance and training stability should degrade sharply when \(r<6\), while gains should saturate for \(r\ge6\).

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a causal order-2 input/output filter lift with an SVD bottleneck and quadratic critic-regression MVP. Noiseless SIMO/MIMO ranks matched the predictions 4 and 6 exactly; noisy tail singular values scaled linearly with observation noise, and the cutoff transitioned from rank 6 to 8 across the noise floor. The bottleneck reduced the feature condition number dramatically, but critic test MSE was essentially unchanged, so the observed benefit is numerical stabilization rather than a clear task-performance win.

### Mechanism check

- Verdict: Built a causal order-2 input/output filter lift with an SVD bottleneck and quadratic critic-regression MVP. Noiseless SIMO/MIMO ranks matched the predictions 4 and 6 exactly; noisy tail singular values scaled linearly with observation noise, and the cutoff transitioned from rank 6 to 8 across the noise floor. The bottleneck reduced the feature condition number dramatically, but critic test MSE was essentially unchanged, so the observed benefit is numerical stabilization rather than a clear task-performance win.
- Confidence: 8/10
- Limitations: This is a synthetic linear-system verification, not a full actor-critic, SAC, or closed-loop policy-learning experiment. The filter bank uses fixed order-2 dynamics, only one main seed is reported for critic regression, and Procrustes alignment, online SVD updates, compute speed, and plant/model mismatch were not tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.8938
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/859/bench_report.json)
- [experiment.py](https://synthcore.org/code/859/experiment.py)
- [report.md](https://synthcore.org/code/859/report.md)
- [report_bench_2026-09-01T125339.md](https://synthcore.org/code/859/report_bench_2026-09-01T125339.md)
- [stage2_bench.py](https://synthcore.org/code/859/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/859)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.