# Rank-One Feedback Spectrum Regularizer

- ID: 381
- Canonical URL: https://synthcore.org/idea/381/rank-one-feedback-spectrum-regularizer
- API JSON: https://synthcore.org/api/idea/381.json
- API Markdown: https://synthcore.org/api/idea/381.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.02877](https://arxiv.org/abs/2607.02877)
- Category: dynamics
- Solves: stability, generalization, accuracy
- ML areas: ssm, rnn, training-dynamics, regularization
- Math tags: dynamical-systems, control-theory, linear-algebra
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Model the scalar feedback route in a recurrent layer as a rank-one perturbation of its open-loop transition. Regularize the frequency response of that route so that no mode reaches unit loop gain, directly targeting oscillatory and slowly decaying instabilities rather than relying only on gradient clipping.

## Mathematical statement

The paper linearizes around a stationary solution and obtains a finite-memory renewal equation with characteristic equation E(λ)=1; the roots determine the feedback spectrum and stability. It also states the zero-discount identity E(0)=Φ′(E*)=B(0), linking closure resonance to loop gain. For a neural linearization, let δx_{k+1}=Aδx_k+bδE_k and δE_k=cᵀδx_k, where A is the open-loop Jacobian, b is the state response to scalar feedback, c is the scalar readout, and J=A+bcᵀ is the closed-loop Jacobian. The matrix determinant lemma gives det(zI−J)=det(zI−A)[1−cᵀ(zI−A)⁻¹b]. Therefore closed-loop modes satisfy the transferred characteristic condition E(z):=cᵀ(zI−A)⁻¹b=1. Keeping |E(z)| below one on a contour around the unit circle is a practical sufficient margin, though not a complete stability theorem for arbitrary nonnormal A.

## Key formulas

- $$\mathcal E(\lambda)=1,$$
- $$\mathcal E(0)=\Phi'(E^*)=B(0),$$
- $$\delta x_{k+1}=A\,\delta x_k+b\,\delta E_k,\qquad \delta E_k=c^\top\delta x_k,\qquad J=A+bc^\top,$$
- $$\det(zI-J)=\det(zI-A)\left[1-c^\top(zI-A)^{-1}b\right],\qquad \mathcal E(z):=c^\top(zI-A)^{-1}b.$$

## Implementation notes

(1) Add this to the transition core of an RNN or diagonal-plus-low-rank SSM. During training, obtain A, b, and c at minibatch states using automatic differentiation: A is the Jacobian with respect to the hidden state, b is the derivative of the next state with respect to the scalar feedback, and c is the derivative of the feedback with respect to the state. Do not materialize J. (2) For probe points z=rho exp(i theta), solve `(z*I-A) q=b`, compute H=dot(c,q), and accumulate `penalty += relu(abs(H)-(1-margin))**2`. Use 8 to 16 theta values and two or three radii near the unit circle; detach the penalty for most minibatches if cost is excessive. (3) The mathematical quantity is H=E(z)=cᵀ(zI−A)⁻¹b. Estimate it with linear solves, while independently materializing J on a small model to compare the predicted gain with its largest eigenvalue and impulse-response decay. At z=1, estimate the zero-frequency loop gain cᵀ(I−A)⁻¹b when the solve is well conditioned. (4) First test a 64-dimensional SSM on sequential MNIST and permuted sequential MNIST against the same model without the penalty, gradient clipping, and spectral normalization. Measure long-context accuracy, gradient norms, impulse-response decay, and training loss oscillations at equal FLOPs. The expected signal is reduced oscillation and improved stability near the largest usable step size without hurting memory-task accuracy. Falsify the idea if the frequency response does not predict observed instability or if the regularizer consistently damages task performance.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a reproducible rank-one feedback spectrum regularizer using complex linear solves over a contour, plus an independent determinant-lemma check. The identity held with maximum relative error 5.8e-15 and closed-loop characteristic residual 8.0e-14. In the scalar toy, the baseline reached g=0.1800 with pole 1.1000, contour gain 3.00, and 19-step impulse magnitude 6.12; the regularized model reached g=0.0650 with pole 0.9850, contour gain 1.084, and impulse magnitude 0.751, but its task MSE increased from approximately 0 to 0.0132. This is a clear stability signal, not evidence of improved task accuracy.

### Mechanism check

- Verdict: Built a reproducible rank-one feedback spectrum regularizer using complex linear solves over a contour, plus an independent determinant-lemma check. The identity held with maximum relative error 5.8e-15 and closed-loop characteristic residual 8.0e-14. In the scalar toy, the baseline reached g=0.1800 with pole 1.1000, contour gain 3.00, and 19-step impulse magnitude 6.12; the regularized model reached g=0.0650 with pole 0.9850, contour gain 1.084, and impulse magnitude 0.751, but its task MSE increased from approximately 0 to 0.0132. This is a clear stability signal, not evidence of improved task accuracy.
- Confidence: 8/10
- Limitations: Only a scalar analytical toy was trained; no minibatch autodiff Jacobians, 64-dimensional RNN/SSM, sequential or permuted MNIST, gradient-clipping comparison, spectral normalization comparison, FLOP accounting, CUDA execution, or nonnormal-A stress test was performed. The contour margin was not fully achieved because the chosen toy optimum and penalty weight leave a small residual violation.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 1
- Paired wins: 0/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/1280/bench_report.json)
- [experiment.py](https://synthcore.org/code/1280/experiment.py)
- [report.md](https://synthcore.org/code/1280/report.md)
- [report_bench_2026-09-04T112349.md](https://synthcore.org/code/1280/report_bench_2026-09-04T112349.md)
- [results.json](https://synthcore.org/code/1280/results.json)
- [stage2_bench.py](https://synthcore.org/code/1280/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1280)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.