# Cone Bi-Rayleigh Stability Regularizer

- ID: 274
- Canonical URL: https://synthcore.org/idea/274/cone-bi-rayleigh-stability-regularizer
- API JSON: https://synthcore.org/api/idea/274.json
- API Markdown: https://synthcore.org/api/idea/274.md
- Verification status: mechanism_failed
- Source: [arXiv:2606.31129](https://arxiv.org/abs/2606.31129)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rnn, ssm, training-dynamics, regularization
- Math tags: spectral-theory, optimization, convex-analysis, dynamical-systems
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Add a two-sided cone-restricted spectral penalty to a recurrent or state-space model. Instead of estimating growth using a symmetric singular-value surrogate, jointly optimize a positive right vector and positive left vector in the extended quotient from the paper, targeting a real generalized eigenvalue of the learned non-selfadjoint transition operator.

## Mathematical statement

The paper studies the generalized spectral equation \(\mathcal{L}u=\lambda\mathcal{G}u\), where \(\mathcal{L}\) is possibly non-selfadjoint and \(\mathcal{G}\) may be singular. For an admissible cone pair \((u,v)\), with right vector \(u\) and left/test vector \(v\), it defines the extended two-variable Rayleigh quotient \(\mathcal{R}(u,v)=\langle\mathcal{L}u,v\rangle/(\mathcal{G}u,v)_H\). Here \(\langle\cdot,\cdot\rangle\) is the operator-duality pairing, \((\cdot,\cdot)_H\) is the Hilbert-space inner product, and the denominator must be positive on the admissible pair. At a positive right-left eigenpair satisfying \(\mathcal{L}u=\lambda\mathcal{G}u\) and the corresponding left relation, the quotient equals \(\lambda\), including non-selfadjoint and non-invertible cases. Set \(\mathcal{L}=I-A_\theta\) and \(\mathcal{G}=I\), where \(A_\theta\) is the local recurrent transition Jacobian. Then \(\lambda=1-\mu\), with \(\mu\) a real growth eigenvalue of \(A_\theta\). Cone constraints \(u,v\ge 0\) are enforced by softplus parameterizations.

## Key formulas

- $$\mathcal{L}u=\lambda\mathcal{G}u$$
- $$\mathcal{R}(u,v)=\frac{\langle\mathcal{L}u,v\rangle}{(\mathcal{G}u,v)_H}$$
- $$\mathcal{L}=I-A_\theta,\qquad \mathcal{G}=I,\qquad \widehat{\lambda}(u,v)=\frac{\langle (I-A_\theta)u,v\rangle}{\langle u,v\rangle+\varepsilon},\qquad \widehat{\mu}=1-\widehat{\lambda}$$
- $$\mathcal{L}u-\widehat{\lambda}\mathcal{G}u\approx 0,\qquad \mathcal{L}^*v-\widehat{\lambda}\mathcal{G}^*v\approx 0$$

## Implementation notes

Integrate this into the training step of a tanh RNN, state-space model, or implicit residual layer with hidden update \(h_{t+1}=F_\theta(h_t,x_t)\). The exact integration point is the local transition Jacobian \(A_\theta=\partial F_\theta/\partial h\), accessed through Jacobian-vector products rather than materialized. Initialize unconstrained probe parameters \(a,b\), set \(u=\operatorname{softplus}(a)+\delta\) and \(v=\operatorname{softplus}(b)+\delta\), and normalize both by their Euclidean norms. Compute \(Au\) using a forward-mode JVP and \(A^\top v\) using a reverse-mode VJP. Evaluate \(n=\langle u-Au,v\rangle\), \(d=\langle u,v\rangle+\varepsilon\), and \(\lambda=n/d\). Perform 3-10 alternating probe updates minimizing the right and left residuals \(r_R=\|u-Au-\lambda u\|_2^2\) and \(r_L=\|v-A^\top v-\lambda v\|_2^2\), then detach the probes before the outer model update. Add \(\rho\operatorname{softplus}(\mu-\mu_{\max})^2\) to the task loss, with \(\mu=1-\lambda\). Reject probes when \(d<10\varepsilon\). The quotient and residual equations come from the paper; softplus, thresholds, probe refresh rate, and penalty weight are empirical. First test on adding-task and copy-memory benchmarks using a 128-unit tanh RNN. Compare vanilla training, spectral-norm regularization, and this method at matched JVP/VJP cost. Track long-horizon loss, exploding-gradient frequency, residual norms, and wall-clock overhead. Success is lower long-horizon error and fewer unstable runs at equal parameter count; failure is no residual convergence or no improvement over spectral normalization.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a cone bi-Rayleigh quotient regularizer prototype and verified the core math numerically: at a positive right/left eigenpair, the quotient matched 1−mu to 2.6e-12, and alternating positive probe updates reduced the joint residual by about 1e24x. In the RNN mini-experiment, cone regularization improved mean test loss over vanilla (0.0306 vs 0.0867) but was substantially slower (18.0s vs 8.4s) and clearly worse than spectral regularization (0.00410 test loss, 10.6s); therefore it did not demonstrate a win over the standard control.

### Mechanism check

- Verdict: Built a cone bi-Rayleigh quotient regularizer prototype and verified the core math numerically: at a positive right/left eigenpair, the quotient matched 1−mu to 2.6e-12, and alternating positive probe updates reduced the joint residual by about 1e24x. In the RNN mini-experiment, cone regularization improved mean test loss over vanilla (0.0306 vs 0.0867) but was substantially slower (18.0s vs 8.4s) and clearly worse than spectral regularization (0.00410 test loss, 10.6s); therefore it did not demonstrate a win over the standard control.
- Confidence: 8/10
- Limitations: The mini-task was a small delayed-output toy rather than the requested adding/copy-memory benchmarks, used a materialized recurrent matrix instead of true JVP/VJP calls, did not measure exploding-gradient frequency or matched FLOPs, and used only three seeds. The cone penalty was often nearly inactive because the estimated growth eigenvalue was below the threshold.

## Artifacts

- [experiment.py](https://synthcore.org/code/1248/experiment.py)
- [report.md](https://synthcore.org/code/1248/report.md)
- [results.json](https://synthcore.org/code/1248/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1248)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.