# Jacobian-Cocycle Growth Controller

- ID: 629
- Canonical URL: https://synthcore.org/idea/629/jacobian-cocycle-growth-controller
- API JSON: https://synthcore.org/api/idea/629.json
- API Markdown: https://synthcore.org/api/idea/629.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.09314](https://arxiv.org/abs/2607.09314)
- Category: dynamics
- Solves: stability, generalization
- ML areas: rnn, ssm, training-dynamics, optimizer
- Math tags: dynamical-systems, spectral-theory, linear-algebra, control-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Treat the sequence of recurrent or state-space Jacobians along a trajectory as a noncommutative matrix cocycle, analogous to the time-dependent offspring mean matrices in the branching model. Estimate its finite-horizon growth exponent and use it to adapt spectral normalization or recurrent gain, targeting a slightly negative exponent for stable memory without uncontrolled exploding dynamics.

## Mathematical statement

Let the neural state update be h_(t+1) = F_theta(h_t, u_t), and let J_t = partial F_theta(h_t, u_t) / partial h_t be its d by d state Jacobian. The propagation over n steps is the ordered product P_(t,n) = J_(t+n-1) ... J_t. The branching-process analogue uses products of nonnegative mean reproduction matrices along the environment, with row-sum norm ||A||_infinity = max_i sum_j |a_(i,j)| and spectral radius rho(A). Define the finite-horizon cocycle growth exponent gamma_t(n) = n^(-1) log ||P_(t,n)||. When the asymptotic limit exists, gamma = lim_(n to infinity) gamma_t(n). A negative exponent implies exponential contraction, a positive exponent implies exponential amplification, and gamma = 0 is the critical boundary. Because products are noncommutative, constraining each rho(J_t) or ||J_t|| separately is only a sufficient condition and can miss cancellation or transient amplification. The controller estimates gamma over a short horizon and enforces a target gamma_target below zero using a differentiable penalty or multiplicative gain correction.

## Key formulas

- $$||A||_infinity = max_(1 <= i <= d) sum_(j=1)^d |a_(i,j)|,qquad ||AB||_infinity <= ||A||_infinity ||B||_infinity$$
- $$P_(t,n) = J_(t+n-1) ... J_t,\qquad gamma_t(n) = (1/n) log ||P_(t,n)||_infinity$$
- $$gamma = limsup_(n to infinity) (1/n) log ||J_(n-1) ... J_0||_infinity,\qquad gamma < 0: contracting,\quad gamma = 0: critical,\quad gamma > 0: amplifying$$
- $$L_cocycle = [max(0, gamma_hat_H - gamma_target)]^2,\qquad gamma_hat_H = (1/H) log ||J_(t+H-1) ... J_t||_2$$

## Implementation notes

1. Integration point: apply this to an RNN, gated state-space model, or continuous-depth residual block during training. For an RNN h_(t+1) = phi(W_h h_t + W_u u_t + b), compute J_t = Diag(phi'(W_h h_t + W_u u_t + b)) W_h. For a general cell, obtain Jacobian-vector products with automatic differentiation rather than explicitly materializing the Jacobian. 2. Pseudocode: sample a training sequence and unroll H states. Initialize a random unit vector v. At each time t compute v_new = J_t v, record r_t = log(norm(v_new)_2 + epsilon), and renormalize v = v_new / (norm(v_new)_2 + epsilon). Set gamma_hat_H = H^(-1) sum_t r_t. Add lambda times [max(0, gamma_hat_H - gamma_target)] squared to the task loss, or rescale the recurrent gain by g <- g exp(-eta_c (gamma_hat_H - gamma_target)) whenever the estimate is above target. Initially stop gradients through the controller if the optimization becomes unstable. 3. Computed versus estimated: Jacobian-vector products are computed exactly by autodiff, while the full matrix product and asymptotic exponent are estimated from one or several vectors over finite horizon H. Use multiple vectors with QR reorthogonalization if estimating more than the leading exponent. The paper supplies the cocycle growth mechanism and sign criterion; the target margin and controller gain are engineering choices. 4. First cheap experiment: compare a vanilla RNN, per-step spectral normalization, and this controller on sequential MNIST or the adding problem with hidden size 128 and horizons H equal to 20, 50, and 100. Sweep recurrent gain and plot task loss, gradient norm, tangent-perturbation norm, and gamma_hat_H. The falsifiable prediction is a sharp transition near gamma_hat_H = 0: positive-exponent runs should show exponential tangent growth and increasingly frequent exploding gradients, while negative-exponent runs should show decay approximately proportional to exp(n gamma_hat_H). Across gain sweeps, the observed divergence boundary should agree with the estimated zero crossing within 20 percent, and increasing H should stabilize the measured boundary rather than move it arbitrarily.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built and numerically verified the Jacobian-Cocycle Growth Controller. The toy predictions were confirmed: scalar growth crossed at gain 1.000, the noncommutative cocycle zero-growth gain was predicted as 1.05830 versus observed 1.05846, tangent growth matched exactly, and the controller converged to the analytically predicted gain 0.957586 with gamma=-0.10. In the secondary RNN test, the controller reduced the final estimated growth and gradient norm, but test loss was essentially unchanged (0.05273 baseline versus 0.05300 controller), so no task-level win was observed.

### Mechanism check

- Verdict: Built and numerically verified the Jacobian-Cocycle Growth Controller. The toy predictions were confirmed: scalar growth crossed at gain 1.000, the noncommutative cocycle zero-growth gain was predicted as 1.05830 versus observed 1.05846, tangent growth matched exactly, and the controller converged to the analytically predicted gain 0.957586 with gamma=-0.10. In the secondary RNN test, the controller reduced the final estimated growth and gradient norm, but test loss was essentially unchanged (0.05273 baseline versus 0.05300 controller), so no task-level win was observed.
- Confidence: 9/10
- Limitations: The primary verification uses a small synthetic 4-dimensional matrix cocycle and finite horizon H=100, with exact products available; it does not establish asymptotic behavior for arbitrary cocycles. The RNN comparison is only one 32-unit, 40-step adding-like task and one seed, and compares fixed gain rather than a full modern spectral-normalization baseline. No sequential MNIST, multi-vector QR estimator, training-overhead measurement, or broad hyperparameter sweep was tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 1
- Paired wins: 0/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/282/bench_report.json)
- [cocycle_experiment.py](https://synthcore.org/code/282/cocycle_experiment.py)
- [report.md](https://synthcore.org/code/282/report.md)
- [report_bench_2026-08-31T001100.md](https://synthcore.org/code/282/report_bench_2026-08-31T001100.md)
- [results.json](https://synthcore.org/code/282/results.json)
- [rnn_mini_experiment.py](https://synthcore.org/code/282/rnn_mini_experiment.py)
- [rnn_results.json](https://synthcore.org/code/282/rnn_results.json)
- [stage2_cocycle_bench.py](https://synthcore.org/code/282/stage2_cocycle_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/282)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.