✗ Failed on benchmark 2026

Projective-Gap Regularization for Random Jacobian Cocycles

Implementation & benchmark of arXiv:2608.18372 — (co)Quasi-irreducible and (co)expanding random maps

Usefulness7/10

Difficulty5/10

Novelty7/10

Dynamics Stability Accuracy Generalization Rnn Ssm Training dynamics Regularization Dynamical systems Spectral theory Random matrix Geometry

Source paper: (co)Quasi-irreducible and (co)expanding random maps arXiv:2608.18372 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Treat the input- or minibatch-dependent Jacobians of a recurrent or state-space network as a random derivative cocycle, and regularize its second Lyapunov exponent away from the first while independently placing the top exponent in a target stable range. This transfers the paper's equivalence between quasi-irreducibility, projective contraction, and a vertical spectral gap into a measurable training objective and a long-horizon stability monitor.

Formulas

$$P_n=A_{n-1}A_{n-2}\cdots A_0,\qquad \lambda_1=\lim_{n\to\infty}\frac{1}{n}\log\|P_nv\|.$$

$$v_{t+1}=\frac{A_tv_t}{\|A_tv_t\|},\qquad \gamma=\lambda_1-\lambda_2>0\ \Longrightarrow\ d_{\mathbb P}(v_t,w_t)\approx d_{\mathbb P}(v_0,w_0)e^{-\gamma t}.$$

$$\widehat{\mathcal L}=\mathcal L_{\rm task}+\kappa_\gamma[\gamma_\star-(\widehat\lambda_1-\widehat\lambda_2)]_+^2+\kappa_1(\widehat\lambda_1-\lambda_\star)^2,$$

$$\widehat\lambda_1=\frac{1}{K}\sum_{t=1}^{K}\log|R_t^{11}|,\qquad \widehat\lambda_2=\frac{1}{K}\sum_{t=1}^{K}\log|R_t^{22}|.$$

Mathematical statement

Let F_t be the neural-network state update at time t, let z_t be its hidden state, and let A_t = D_z F_t(z_t) be the random Jacobian induced by the input, data sample, or minibatch. The tangent cocycle is P_n = A_{n-1} ... A_0. For a nonzero tangent vector v, the projective dynamics is v_{t+1} = A_t v_t / ||A_t v_t||. The top Lyapunov exponent is lambda_1 = lim_{n to infinity} n^{-1} log ||P_n v||, while lambda_2 is estimated by two-vector QR iteration. The paper's vertical spectral-gap mechanism predicts projective contraction when gamma = lambda_1 - lambda_2 > 0: normalized tangent directions with generic initial conditions converge at an approximate rate exp(-gamma n). In the adapted loss, lambda_1 is constrained to a target lambda_star, and gamma is forced above a margin gamma_star. A negative lambda_star gives contractive dynamics, lambda_star near zero gives memory-preserving dynamics, and a positive value gives controlled expansion.

Implementation notes

1. Integration point: apply this to an RNN, S4-style state-space model, or residual network with hidden update z_{t+1} = F_theta(z_t, x_t). During training, retain a short sequence of hidden states and estimate Jacobian-vector products using automatic differentiation; do not materialize the full Jacobian. 2. Pseudocode: sample a sequence or minibatch; initialize two orthonormal tangent vectors q_1 and q_2; for each of K steps compute u_i = JVP(F_theta, z_t, q_i), stack the vectors, perform differentiable QR, replace q_i by the QR columns, and accumulate log absolute values of the diagonal entries R_t^{ii}. Set lambda_hat_i equal to K^{-1} times the accumulated log diagonal terms, add the displayed gap and top-exponent penalties to the task loss, and backpropagate. Use stop-gradient through QR initially if differentiability is unstable, and update estimates with an exponential moving average. 3. Computed versus estimated: the projective update and QR identities are the random-cocycle mechanism from the paper; lambda_1, lambda_2, and gamma are empirical finite-window estimates. The simple-top-exponent and no-equator assumptions are not known beforehand, so test them by repeating the estimate from many random tangent bases and measuring convergence of projective directions. 4. First cheap experiment: train a 2-layer tanh RNN on the adding problem with sequence lengths 100 and 500, comparing ordinary Adam against the regularizer with lambda_star = 0 and gamma_star = 0.05. Use K = 50 Jacobian steps per minibatch. The falsifiable signature is exponential decay of the angle between two independently initialized normalized tangent vectors with fitted slope approximately equal to -gamma_hat; the measured slope should agree with -(lambda_hat_1 - lambda_hat_2) within 20 percent. Runs with a negative gap should show persistent direction dependence and larger long-horizon gradient variance. Sweep kappa_gamma and verify a transition near the point where the estimated gap crosses zero.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a reproducible projective-gap cocycle verifier and a small tanh-RNN integration experiment. The mechanism manifested quantitatively: fitted projective-decay slopes matched -gamma within 0.5%-6.9%, and threshold-hit times matched log(theta0/epsilon)/gamma within 2.7%-3.2%; at zero gap, projective slopes stayed near zero. The RNN regularizer produced a modest lower final task loss (1182.1 vs 1225.6), but its measured lambda1 (-0.805) and gap (1.674) overshot the requested targets, so this is evidence for the mechanism rather than a tuned training win.

Agent confidence: 8/10

Baseline: RNN final loss 1225.576 over the final 20 iterations
Idea: RNN final loss 1182.060; estimated lambda1=-0.805, gap=1.674; toy gap-sweep slope relative error 0.5%-6.9%; threshold-time ratio 0.973-1.032

Limitations: The cocycle verifier uses a deliberately controlled 2D diagonal cocycle, not a learned nonlinear network. The RNN comparison is one seed, one sequence length, and a short run; baseline and regularized runs use separate random streams, and the regularizer uses explicit small-matrix SVDs rather than production JVP/QR estimation. No adding-task generalization, multiple-seed statistics, or kappa sweep was performed.

How to run: python3 projective_gap_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.18372) Sep 2, 2026

Ahead of the baseline (−7.1%) but not statistically significant (p=0.1008, wins 6 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0404
Idea mean: 0.0375
Effect (Δ): -0.0029 (−7.1%; negative = idea better)
Wins: 6 / 8 paired seeds
p-value: 0.1008 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

The benchmark used 4 epochs and 400 training/400 test samples to fit the execution budget; no longer training schedule, larger model, or additional task tracks were tested. The signature used short-window trained-GRU JVP behavior and did not meet the 30% quantitative tolerance.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented differentiable two-vector JVP/QR projective-gap regularization for the shared rnn_small GRU on the structurally matched dynamics track. The best idea configuration had lower mean test MSE than the tuned baseline (0.037517 versus 0.040376), but the paired permutation test was not significant (delta=-0.002859, p=0.10075), so there is no validated training win. The trained-model mechanism signature also failed quantitatively: observed projective slope -0.03503 versus predicted -0.08474, relative error 58.7%.

Baseline: dynamics/rnn_small, 8 seeds, best lr=0.002, mean test MSE=0.0403763, std=0.0196119
Idea: dynamics/rnn_small with JVP/QR gap and top-exponent penalty, 8 seeds, best lr=0.002 and kappa=0.05, mean test MSE=0.0375170, std=0.0202588, paired delta=-0.0028594, permutation p=0.10075
Mechanism signature: map[confirmed:false observed_gap:0.08474337961524725 observed_log_angle_slope:-0.035032859627017375 predicted_minus_gap:-0.08474337961524725 relative_error:0.5866006313876798]

Benchmark result