# Spectral-Margin Loop Regularizer

- ID: 701
- Canonical URL: https://synthcore.org/idea/701/spectral-margin-loop-regularizer
- API JSON: https://synthcore.org/api/idea/701.json
- API Markdown: https://synthcore.org/api/idea/701.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.10681](https://arxiv.org/abs/2607.10681)
- Category: regularization
- Solves: stability, accuracy, speedup
- ML areas: transformer, training-dynamics, optimizer
- Math tags: spectral-theory, linear-algebra, dynamical-systems, stability
- Ratings: usefulness 7/10; difficulty 7/10; novelty 7/10

## Idea description

Regularize the local recurrent Jacobian by its spectral radius rather than imposing the overly conservative operator-norm condition $\|J\|_2<1$. This permits useful non-normal updates with transient amplification while explicitly pushing the asymptotic dynamics toward a stable fixed point.

## Mathematical statement

For $h_{t+1}=\bar{A}\odot h_t+f(h_t,e)$, define the local Jacobian $J_t=\operatorname{Diag}(\bar{A})+\partial f(h_t,e)/\partial h_t$, where $\operatorname{Diag}(\bar{A})$ is the diagonal carry operator and the derivative is with respect to the flattened state $h_t$. The spectral radius is $\rho(J_t)=\max_i|\lambda_i(J_t)|$, where $\lambda_i$ are eigenvalues. Non-normal matrices can satisfy $\|J_t\|_2>1$ while $\rho(J_t)<1$, so an operator-norm penalty can reject asymptotically stable but expressive loops. Use $L_{\mathrm{spec}}=\operatorname{ReLU}(\rho(J_t)-r)^2$ with target margin $0<r<1$. Optionally control finite transient amplification with $L_{\mathrm{trans}}=\operatorname{ReLU}(\max_{1\leq k\leq K}\|J_t^k\|_2-g)^2$, where $K$ is a short horizon and $g$ is an allowed gain.

## Key formulas

- $$J_t=\operatorname{Diag}(\bar{A})+\frac{\partial f(h_t,e)}{\partial h_t}$$
- $$\rho(J_t)=\max_i|\lambda_i(J_t)|,\qquad \rho(J_t)\leq r<1$$
- $$L_{\mathrm{spec}}=\big[\max(0,\rho(J_t)-r)\big]^2$$
- $$L_{\mathrm{trans}}=\left[\max\left(0,\max_{1\leq k\leq K}\|J_t^k\|_2-g\right)\right]^2$$

## Implementation notes

Add the regularizer to a tied recurrent transformer during training, using the Jacobian of one or a few sampled loop states rather than unrolling every state. At each optimizer step, select a minibatch and loop index t, flatten h_t, and define a Jacobian-vector product function v -> J_t v using automatic differentiation through the update h_next=A*h_t+Block(h_t+e)-(h_t+e). For small d or short sequences, explicitly materialize J_t and compute eigenvalues with eigvals; for realistic models use a blockwise approximation and 8-16 Arnoldi iterations to estimate the largest-magnitude eigenvalue. Set r=0.95 initially and add lambda_spec*[max(0,rho-r)]^2 to the task loss. To avoid severe transient explosions, optionally estimate ||J^k||_2 by power iteration on K=3 powers of the JVP and add the transient term with g=2 or 4. Backpropagate through the differentiable eigensolver when available; otherwise stop gradients through the estimated rho and use it as a controller that increases lambda_spec when rho exceeds r. The paper's mathematical quantity is rho(J); the empirical estimate is the Arnoldi or power-iteration approximation. First test a 4-loop, 2-layer tied transformer on a delayed-copy task and CIFAR-10, comparing no regularizer, operator-norm regularization, and spectral regularization at equal training FLOPs. Measure loss stability, gradient spikes, final accuracy, estimated rho, and transient amplification. The desired result is lower divergence frequency and better accuracy than norm control, while allowing some ||J||_2>1.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Implemented the spectral-radius regularizer and verified its mechanism on non-normal Jordan loops. The spectral radius stayed at 0.8 across shear values, observed norm scaling was 0.954 versus predicted 1, observed transient-gain scaling was 2.040 versus predicted 2, and the scalar stability-boundary sweep was 100% correct. The learned comparison showed spectral control allowing norm 1.503 with rho 0.878, but it did not outperform operator control on final MSE.

### Mechanism check

- Verdict: Implemented the spectral-radius regularizer and verified its mechanism on non-normal Jordan loops. The spectral radius stayed at 0.8 across shear values, observed norm scaling was 0.954 versus predicted 1, observed transient-gain scaling was 2.040 versus predicted 2, and the scalar stability-boundary sweep was 100% correct. The learned comparison showed spectral control allowing norm 1.503 with rho 0.878, but it did not outperform operator control on final MSE.
- Confidence: 8/10
- Limitations: Only a tiny 2D linear recurrent learner was tested; no tied recurrent transformer, delayed-copy task, or CIFAR-10. The toy used an exact eigensolver rather than Arnoldi/JVP estimation, and the transient term was measured but not trained.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0163
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/301/bench_report.json)
- [report.md](https://synthcore.org/code/301/report.md)
- [report_bench_2026-08-31T005703.md](https://synthcore.org/code/301/report_bench_2026-08-31T005703.md)
- [results.json](https://synthcore.org/code/301/results.json)
- [spectral_margin_bench.py](https://synthcore.org/code/301/spectral_margin_bench.py)
- [spectral_margin_experiment.py](https://synthcore.org/code/301/spectral_margin_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/301)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.