# Projective-Gap Regularization for Random Jacobian Cocycles

- ID: 2518
- Canonical URL: https://synthcore.org/idea/2518/projective-gap-regularization-for-random-jacobian-cocycles
- API JSON: https://synthcore.org/api/idea/2518.json
- API Markdown: https://synthcore.org/api/idea/2518.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.18372](https://arxiv.org/abs/2608.18372)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rnn, ssm, training-dynamics, regularization
- Math tags: dynamical-systems, spectral-theory, random-matrix, geometry
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Treat the input- or minibatch-dependent Jacobians of a recurrent or state-space network as a random derivative cocycle, and regularize its second Lyapunov exponent away from the first while independently placing the top exponent in a target stable range. This transfers the paper's equivalence between quasi-irreducibility, projective contraction, and a vertical spectral gap into a measurable training objective and a long-horizon stability monitor.

## Mathematical statement

Let F_t be the neural-network state update at time t, let z_t be its hidden state, and let A_t = D_z F_t(z_t) be the random Jacobian induced by the input, data sample, or minibatch. The tangent cocycle is P_n = A_{n-1} ... A_0. For a nonzero tangent vector v, the projective dynamics is v_{t+1} = A_t v_t / ||A_t v_t||. The top Lyapunov exponent is lambda_1 = lim_{n to infinity} n^{-1} log ||P_n v||, while lambda_2 is estimated by two-vector QR iteration. The paper's vertical spectral-gap mechanism predicts projective contraction when gamma = lambda_1 - lambda_2 > 0: normalized tangent directions with generic initial conditions converge at an approximate rate exp(-gamma n). In the adapted loss, lambda_1 is constrained to a target lambda_star, and gamma is forced above a margin gamma_star. A negative lambda_star gives contractive dynamics, lambda_star near zero gives memory-preserving dynamics, and a positive value gives controlled expansion.

## Key formulas

- $$P_n=A_{n-1}A_{n-2}\cdots A_0,\qquad \lambda_1=\lim_{n\to\infty}\frac{1}{n}\log\|P_nv\|.$$
- $$v_{t+1}=\frac{A_tv_t}{\|A_tv_t\|},\qquad \gamma=\lambda_1-\lambda_2>0\ \Longrightarrow\ d_{\mathbb P}(v_t,w_t)\approx d_{\mathbb P}(v_0,w_0)e^{-\gamma t}.$$
- $$\widehat{\mathcal L}=\mathcal L_{\rm task}+\kappa_\gamma[\gamma_\star-(\widehat\lambda_1-\widehat\lambda_2)]_+^2+\kappa_1(\widehat\lambda_1-\lambda_\star)^2,$$
- $$\widehat\lambda_1=\frac{1}{K}\sum_{t=1}^{K}\log|R_t^{11}|,\qquad \widehat\lambda_2=\frac{1}{K}\sum_{t=1}^{K}\log|R_t^{22}|.$$

## Implementation notes

1. Integration point: apply this to an RNN, S4-style state-space model, or residual network with hidden update z_{t+1} = F_theta(z_t, x_t). During training, retain a short sequence of hidden states and estimate Jacobian-vector products using automatic differentiation; do not materialize the full Jacobian. 2. Pseudocode: sample a sequence or minibatch; initialize two orthonormal tangent vectors q_1 and q_2; for each of K steps compute u_i = JVP(F_theta, z_t, q_i), stack the vectors, perform differentiable QR, replace q_i by the QR columns, and accumulate log absolute values of the diagonal entries R_t^{ii}. Set lambda_hat_i equal to K^{-1} times the accumulated log diagonal terms, add the displayed gap and top-exponent penalties to the task loss, and backpropagate. Use stop-gradient through QR initially if differentiability is unstable, and update estimates with an exponential moving average. 3. Computed versus estimated: the projective update and QR identities are the random-cocycle mechanism from the paper; lambda_1, lambda_2, and gamma are empirical finite-window estimates. The simple-top-exponent and no-equator assumptions are not known beforehand, so test them by repeating the estimate from many random tangent bases and measuring convergence of projective directions. 4. First cheap experiment: train a 2-layer tanh RNN on the adding problem with sequence lengths 100 and 500, comparing ordinary Adam against the regularizer with lambda_star = 0 and gamma_star = 0.05. Use K = 50 Jacobian steps per minibatch. The falsifiable signature is exponential decay of the angle between two independently initialized normalized tangent vectors with fitted slope approximately equal to -gamma_hat; the measured slope should agree with -(lambda_hat_1 - lambda_hat_2) within 20 percent. Runs with a negative gap should show persistent direction dependence and larger long-horizon gradient variance. Sweep kappa_gamma and verify a transition near the point where the estimated gap crosses zero.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built a reproducible projective-gap cocycle verifier and a small tanh-RNN integration experiment. The mechanism manifested quantitatively: fitted projective-decay slopes matched -gamma within 0.5%-6.9%, and threshold-hit times matched log(theta0/epsilon)/gamma within 2.7%-3.2%; at zero gap, projective slopes stayed near zero. The RNN regularizer produced a modest lower final task loss (1182.1 vs 1225.6), but its measured lambda1 (-0.805) and gap (1.674) overshot the requested targets, so this is evidence for the mechanism rather than a tuned training win.

### Mechanism check

- Verdict: Built a reproducible projective-gap cocycle verifier and a small tanh-RNN integration experiment. The mechanism manifested quantitatively: fitted projective-decay slopes matched -gamma within 0.5%-6.9%, and threshold-hit times matched log(theta0/epsilon)/gamma within 2.7%-3.2%; at zero gap, projective slopes stayed near zero. The RNN regularizer produced a modest lower final task loss (1182.1 vs 1225.6), but its measured lambda1 (-0.805) and gap (1.674) overshot the requested targets, so this is evidence for the mechanism rather than a tuned training win.
- Confidence: 8/10
- Limitations: The cocycle verifier uses a deliberately controlled 2D diagonal cocycle, not a learned nonlinear network. The RNN comparison is one seed, one sequence length, and a short run; baseline and regularized runs use separate random streams, and the regularizer uses explicit small-matrix SVDs rather than production JVP/QR estimation. No adding-task generalization, multiple-seed statistics, or kappa sweep was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.10075
- Paired wins: 6/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/968/bench_report.json)
- [projective_gap_experiment.py](https://synthcore.org/code/968/projective_gap_experiment.py)
- [report.md](https://synthcore.org/code/968/report.md)
- [report_bench_2026-09-01T172658.md](https://synthcore.org/code/968/report_bench_2026-09-01T172658.md)
- [results.json](https://synthcore.org/code/968/results.json)
- [stage2_bench.py](https://synthcore.org/code/968/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/968)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.