△ Mechanism confirmed, baseline not beaten 2026

Intrinsic-Rank Filter Memory for Actor-Critic

Implementation & benchmark of arXiv:2608.11750 — Reinforcement Learning-Based Output Feedback LQR for Continuous-Time MIMO Systems

Usefulness7/10

Difficulty5/10

Novelty6/10

Architecture Stability Sample efficiency Scalability Rl Rnn World model Training dynamics Control theory Dynamical systems Linear algebra Optimization

Source paper: Reinforcement Learning-Based Output Feedback LQR for Continuous-Time MIMO Systems arXiv:2608.11750 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace an oversized recurrent hidden state or raw history stack with a causal filtered input-output lift followed by an SVD-selected bottleneck. The actor, critic, and Bellman regression operate only on the identifiable memory coordinates, preventing deterministic null directions from being fitted as if they were independent state variables.

Formulas

$$r_{\mathrm{int}}=\begin{cases}2n,&m=1\\n(m+1),&\text{general MIMO},\end{cases}$$

$$Z=[z(t_1),\ldots,z(t_N)]=U\Sigma V^{\top},\qquad h_r(t)=U_r^{\top}z(t),\quad r=\max\{i:\sigma_i/\sigma_1>\tau\},$$

$$u(t)=\pi_\theta(h_r(t)),\qquad V_\phi(h_r)=h_r^{\top}P_\phi h_r,$$

$$\kappa(\Phi_r)=\frac{\sigma_{\max}(\Phi_r)}{\sigma_{\min}(\Phi_r)},\qquad \Phi_r=\text{Bellman-regression features constructed from }h_r.$$

Mathematical statement

For the continuous-time plant $\dot{x}=Ax+Bu,\ y=Cx$, construct a measurable filtered-history vector $z(t)\in\mathbb{R}^{d}$ from causal stable filters driven by $u$ and $y$. Although $d$ may be large, the paper shows that the intrinsic dimension is $r_{\mathrm{int}}=2n$ for SIMO systems and $r_{\mathrm{int}}=n(m+1)$ for general MIMO systems, where $n$ is the hidden state dimension and $m$ is the input dimension. Given a data matrix $Z=[z(t_1),\ldots,z(t_N)]$, compute $Z=U\Sigma V^\top$ and retain $U_r$, the left singular vectors with $\sigma_i/\sigma_1>\tau$. The neural memory is $h_r(t)=U_r^\top z(t)$. A linear-quadratic critic uses $V_\theta(h_r)=h_r^\top P_\theta h_r$, while a neural critic or actor receives $h_r$ instead of $z$. The useful property is removal of deterministic null directions: the Bellman feature matrix has numerical rank approximately $r_{\mathrm{int}}$, rather than the ambient filter dimension $d$.

Implementation notes

1. Integration point: use a fixed bank of stable first- or second-order continuous-time filters driven by the measured output $y_t$, action $u_t$, and optionally their derivatives estimated by finite differences. Discretize each filter, concatenate its states into $z_t$, and insert the bottleneck $h_t=U_r^\top z_t$ immediately before the actor and critic. For a recurrent neural policy, feed $h_t$ to a small GRU rather than feeding the full filter vector. 2. Pseudocode: collect an exploration rollout; update each filter with $z_{t+1}=A_fz_t+B_f[y_t;u_t]$; form $Z$; compute its SVD; choose $r$ using $\sigma_i/\sigma_1>\tau$, with $\tau=10^{-3}$ initially; set $h_t=U_r^\top z_t$; evaluate $a_t=\pi_\theta(h_t)$; fit the critic from TD targets $q_t+\gamma V_\phi(h_{t+1})$; update the actor with the critic gradient. Recompute $U_r$ only between rollout epochs and align new singular vectors by Procrustes rotation to avoid artificial jumps. 3. Computed from the paper: the target intrinsic-rank formulas $2n$ for SIMO and $n(m+1)$ for MIMO, and the use of reduced filtered coordinates in Bellman iteration. Estimated empirically: the SVD spectrum, numerical rank, Bellman feature matrix $\Phi_r$, and condition number. 4. First cheap experiment: use a two-state continuous-time MIMO plant with $n=2,m=2$, so the predicted intrinsic rank is $r_{\mathrm{int}}=6$, while deliberately constructing a 20- to 40-dimensional filter lift. Compare SAC or deterministic policy gradient with the full lift, the SVD bottleneck at $r=6$, and an incorrectly compressed $r<6$ model. Measure TD residual variance, $\kappa(\Phi_r)$, critic loss resets, and closed-loop regulation cost. The quantitative prediction is that singular values after the sixth remain near numerical noise, $\operatorname{rank}(Z)\approx6$, and the reduced model should have at least an order-of-magnitude smaller condition number than the full regression. Performance and training stability should degrade sharply when $r<6$, while gains should saturate for $r\ge6$.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a causal order-2 input/output filter lift with an SVD bottleneck and quadratic critic-regression MVP. Noiseless SIMO/MIMO ranks matched the predictions 4 and 6 exactly; noisy tail singular values scaled linearly with observation noise, and the cutoff transitioned from rank 6 to 8 across the noise floor. The bottleneck reduced the feature condition number dramatically, but critic test MSE was essentially unchanged, so the observed benefit is numerical stabilization rather than a clear task-performance win.

Agent confidence: 8/10

Baseline: Full 8D noisy lift: test MSE 6.51326e-3; quadratic-feature condition 4.484e7.
Idea: Rank-6 SVD bottleneck: test MSE 6.52388e-3; condition 5.860e1. SIMO observed rank 4/ambient 6, MIMO observed rank 6/ambient 8; noisy tail-to-signal ratio was approximately 0.1964 times noise amplitude.

Limitations: This is a synthetic linear-system verification, not a full actor-critic, SAC, or closed-loop policy-learning experiment. The filter bank uses fixed order-2 dynamics, only one main seed is reported for critic regression, and Procrustes alignment, online SVD updates, compute speed, and plant/model mismatch were not tested.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.11750) Sep 2, 2026

Ahead of the baseline (−3.4%) but not statistically significant (p=0.8938, wins 5 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0011
Idea mean: 0.0011
Effect (Δ): -0 (−3.4%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.8938 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±34.7%
Mechanism: Confirmed ✓
Practical effect: Inconclusive
Baseline tuning: swept over 0 configs

Limitations:

This tests supervised pendulum prediction rather than actor-critic policy learning or Bellman regression. The lift deliberately included duplicated channels to create deterministic null directions; online SVD updates, Procrustes alignment, compute/memory speed, and true closed-loop actor-critic performance were not tested.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Built the intrinsic-rank filter-memory intervention on the structurally matched dynamics track using a canonical rnn_small-style GRU, causal filtering, and train-split SVD bottlenecking. The mechanism signature was confirmed: ambient width 9 reduced to observed rank 6, matching the predicted MIMO intrinsic rank. However, the eight-seed paired result was not significant (delta_mean=-3.8747e-05, p=0.8938), so the idea does not meet the benchmark win criterion.

Baseline: Test MSE 0.0011299664 ± 0.0003363602 over 8 seeds; best lr=0.01.
Idea: Test MSE 0.0010912194 ± 0.0004496559 over 8 seeds; best lr=0.01, tau=0.01; paired delta=-3.8746963e-05, p=0.8938, 5/8 wins.

Benchmark result