# Krylov Resonance Regularization

- ID: 2947
- Canonical URL: https://synthcore.org/idea/2947/krylov-resonance-regularization
- API JSON: https://synthcore.org/api/idea/2947.json
- API Markdown: https://synthcore.org/api/idea/2947.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.28575](https://arxiv.org/abs/2608.28575)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rnn, ssm, training-dynamics, regularization
- Math tags: dynamical-systems, spectral-theory, linear-algebra, control-theory, numerical-analysis
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Add a resonance-estimation module to a recurrent network or state-space model and regularize the decay spectrum of its observable correlations. Instead of using eigenvalues of a small projected recurrent matrix as memory timescales, estimate dominant poles from multi-step correlations and a resolvent/Krylov fit, thereby remaining valid when projection eigenvalues are ill-conditioned or hidden resonances occur. The method is intended to preserve useful long memory while suppressing unstable or excessively slow modes.

## Mathematical statement

Let U be the one-step unitary or approximately norm-preserving hidden-state propagator, A an observable/readout vector, and C(t)=<A,U^tA> its autocorrelation. In finite dimensions, C(t)=sum_j |c_j|^2 exp(i omega_j t), where c_j=<b_j,A> and b_j are eigenvectors of U. For a learned nonunitary recurrent map, replace the exact unitary dynamics by the linearized Jacobian or tangent propagator J_t, and estimate a rational transfer function Rhat(z)=sum_{t=0}^T C(t) z^{-t} whose poles r_k approximate RP resonances. The dominant relaxation time is tau=-1/log|r_*|, where r_* is the largest-modulus pole inside the unit disk. Use a penalty that keeps nuisance poles below 1-epsilon while allowing task-required poles to approach a prescribed memory radius. The paper's key mechanism is that poles of the analytically continued resolvent can differ from eigenvalues of the truncated propagator K_m=Q_m^* U Q_m, particularly when K_m is ill-conditioned because of backflow.

## Key formulas

- $$C(t)=\langle A|U^t|A\rangle,$$
- $$C(t)=\sum_j |c_j|^2e^{i\omega_jt},\qquad c_j=\langle b_j|A\rangle,$$
- $$K_m=Q_m^*UQ_m,\qquad \widehat R(z)=\sum_{t=0}^{T}C(t)z^{-t}\approx \sum_{k=1}^{p}\frac{a_k}{1-r_k z^{-1}},$$
- $$\mathcal L_{\mathrm{res}}=\sum_{k=1}^{p}\operatorname{softplus}\!\left(\frac{|r_k|-(1-\epsilon)}{\tau}\right)^2+\lambda_\tau\bigl(\max(0,\tau_{\min}-\tau_*)\bigr)^2,\qquad \tau_*=-1/\log|r_*|.$$

## Implementation notes

1. Integration point: apply this to a GRU-free linear or nonlinear state-space model, or to an RNN with hidden update h_{t+1}=f_theta(h_t,x_t), using short free-running trajectories. Every M training steps, detach hidden trajectories and compute C(t)=B^{-1} sum_b (v^T h_0^(b))(v^T h_t^(b)) for t=0,...,T, where v is a fixed or randomly refreshed probe vector. 2. Pseudocode: collect correlations; form the Hankel matrices H0[i,j]=C(i+j) and H1[i,j]=C(i+j+1); solve a regularized matrix-pencil problem H1 approximately equal to H0 R; take eigenvalues of R as estimated poles r_k; add lambda_res times the resonance loss to the task loss. Compute the projected recurrent spectrum with Arnoldi only as a diagnostic. 3. Directly computed quantities are correlations, Hankel singular values, pole estimates, and condition number kappa(H0); empirical choices are rank p, probe vectors, epsilon, and target time constant. 4. First experiment: use copy-memory and delayed-MNIST tasks with equal-parameter S4, LRU, and orthogonal-RNN baselines. Compare no regularizer, projected-eigenvalue regularization, and resonance regularization under equal optimizer steps and wall-clock budgets. 5. Preregistered signature: over an intermediate-time window, the correlation envelope must obey |C(t)| approximately equal to c|r_*|^t, and fitted tau_* must predict empirical memory half-life within 20 percent. A conditioning transition is expected when kappa(H0) increases sharply while eigenvalues of K_m disagree with fitted poles. Reject the transfer if fitted poles do not predict memory half-life better than projected eigenvalues, or if the regularizer fails to reduce the fraction of poles with |r_k| greater than or equal to 1 by at least 50 percent without task-loss degradation above 5 percent.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a readable MVP with matrix-pencil/Krylov pole estimation, Hankel conditioning diagnostics, a controlled correlation sanity check, and a delayed-copy RNN experiment. The estimator recovered the true poles 0.93 and 0.62 with maximum modulus error 0.0034, demonstrating the core fitting mechanism. However, the proposed regularization worsened task MSE (0.9456 vs 0.8289) and increased spectral radius (1.0861 vs 1.0636), so no training or stability win was observed.

### Mechanism check

- Verdict: Built a readable MVP with matrix-pencil/Krylov pole estimation, Hankel conditioning diagnostics, a controlled correlation sanity check, and a delayed-copy RNN experiment. The estimator recovered the true poles 0.93 and 0.62 with maximum modulus error 0.0034, demonstrating the core fitting mechanism. However, the proposed regularization worsened task MSE (0.9456 vs 0.8289) and increased spectral radius (1.0861 vs 1.0636), so no training or stability win was observed.
- Confidence: 9/10
- Limitations: Only a tiny nonlinear delayed-copy task was tested; delayed-MNIST, S4/LRU/orthogonal-RNN baselines, wall-clock/FLOP matching, and multi-seed evaluation were not tested. The training penalty uses detached fitted poles to gate a spectral-norm proxy rather than backpropagating through the Hankel pole estimator, and the CUDA-failure fallback was not exercised in the successful run.

## Artifacts

- [README.md](https://synthcore.org/code/1143/README.md)
- [experiment.py](https://synthcore.org/code/1143/experiment.py)
- [report.md](https://synthcore.org/code/1143/report.md)
- [results.json](https://synthcore.org/code/1143/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1143)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.