Idea description

Penalize short positive feedback cycles in an iterative neural module by suppressing products of absolute Jacobian blocks around the cycle. This targets the mechanism responsible for exponential temperature sensitivity rather than merely penalizing the total Jacobian norm, allowing strong feed-forward paths while controlling recurrent amplification.

Formulas

$$\text{DAG susceptibility}=\Theta(\tau^{-(L+1)}),\qquad \text{positive cycle susceptibility}\sim e^{C/\tau}$$

$$G_C=\prod_{a=1}^{r}K_{i_{a+1},i_a},\qquad i_{r+1}=i_1$$

$$\operatorname{tr}(K^r)=\sum_{i_1,\ldots,i_r}K_{i_1i_r}K_{i_ri_{r-1}}\cdots K_{i_2i_1}$$

$$\mathcal{L}_{cycle}=\lambda\sum_{r=2}^{R}\frac{\operatorname{tr}(J^r)}{\tau^r m},\qquad J_{ij}=\left|\frac{\partial h^{i}_{new}}{\partial h^{j}_{old}}\right|$$

Mathematical statement

The paper distinguishes a directed acyclic feedback graph, whose longest path $L$ yields susceptibility $\Theta(\tau^{-(L+1)})$, from a positive cycle, which produces an $e^{C/\tau}$ factor along a Perron direction. For a nonnegative feedback matrix $K$, a directed cycle $C=(i_1,\ldots,i_r,i_1)$ has gain $G_C=\prod_{a=1}^{r}K_{i_{a+1},i_a}$, where $i_{r+1}=i_1$. Its geometric mean $G_C^{1/r}$ measures the per-edge cyclic amplification. For a nonnegative matrix, $\operatorname{tr}(K^r)$ sums products over all closed walks of length $r$, so it is a differentiable surrogate for short positive cycles. In a neural module partitioned into state blocks $h^1,\ldots,h^m$, define $J_{ij}=|\partial h^i_{new}/\partial h^j_{old}|$. The proposed regularizer controls temperature-scaled closed-walk gain through $\operatorname{tr}((J/\tau)^r)$ rather than penalizing every path equally.

Implementation notes

Apply the penalty to a recurrent attention cell, deep equilibrium layer, iterative MoE router, or residual block with explicitly separated state groups. Let $h=(h^1,\ldots,h^m)$ denote block states and estimate the block Jacobian $J_{ij}=|\partial h^i_{new}/\partial h^j_{old}|$ using Hutchinson-style Jacobian products rather than materializing the full matrix. For each minibatch, sample a nonnegative random vector $q$ and compute $u_1=Jq$, $u_2=Ju_1$, up to $u_R=Ju_{R-1}$ with absolute Jacobian-vector products; estimate closed-walk contributions with independent Hutchinson vectors, using $\operatorname{tr}(J^r)\approx q^\top J^r q$ after appropriate normalization. Add $\mathcal{L}_{cycle}=\lambda\sum_{r=2}^{R}\operatorname{tr}(J^r)/(\tau^r m)$ to the task loss, using $R=2$ or $3$ initially. If second-order autodiff is too expensive, compute the Jacobian estimate on detached activations and update the penalty intermittently, or use a finite-difference stop-gradient approximation. For attention, blocks can be heads or token groups; for MoE, they can be experts and router state. The paper supplies the causal-versus-cycle distinction and temperature scaling; the block partition and stochastic trace estimator are engineering choices. Begin with a 4-layer recurrent transformer on CIFAR-10 sequences or WikiText-2, comparing no regularizer, spectral-norm regularization, and this cycle penalty at matched auxiliary-gradient cost. Measure validation loss, gradient spikes, seed variance, effective depth, estimated closed-walk gains, and throughput. The prediction is that cycle regularization reduces instability and seed variance more than a global Jacobian norm penalty at similar task accuracy, while preserving stronger non-cyclic feed-forward interactions.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built and numerically verified a positive-cycle Jacobian penalty MVP for a two-state recurrent cell. The mechanism manifested: tr(K²)=2ab exactly, the growth transition occurred at the predicted gamma*=1/rho=0.833 and was bracketed between 0.76 and 0.90, measured log-growth slopes matched theory exactly, and temperature scaling followed the predicted tau^-2 law. In the mini-experiment, the cycle penalty reduced mean closed-walk gain to 0.0396 versus 0.6008 unregularized and 0.0956 for the spectral proxy, but it did not improve mean task MSE and required about 1.5x the runtime.

Agent confidence: 9/10

Baseline: No regularizer: mean MSE 0.05242 ± 0.03590; mean closed-walk trace2 0.60075; mean runtime 4.40 s
Idea: Cycle penalty: mean MSE 0.05649 ± 0.03337; mean closed-walk trace2 0.03958; mean runtime 7.22 s. Single-seed MSE was 0.02342 versus baseline 0.02345.

Limitations: This is a tiny two-state synthetic regression experiment, not a transformer, attention model, MoE, or deep-equilibrium system. The Jacobian estimate is exact for the toy block structure but uses detached activations, only r=2, three random seeds, and no matched-FLOP larger-scale validation.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.18128) Sep 2, 2026

Ahead of the baseline (−0.0%) but not statistically significant (p=0.2182, wins 7 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0211
Idea mean: 0.0211
Effect (Δ): -0 (−0.0%; negative = idea better)
Wins: 7 / 8 paired seeds
p-value: 0.2182 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.0%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 0 configs

Limitations:

The test used the small built-in dynamics dataset with 200 training and 100 test samples, 3 epochs, two hidden blocks, r=2 only, and tau=1.0. The signature used one representative final-timestep state per trained model. No separate spectral-norm baseline was run; the required standard unregularized baseline was swept over learning rate and weight decay. The mechanism signal is extremely small and should not be treated as strong confirmation.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented the positive-cycle Jacobian penalty on the structurally matched dynamics track using the shared rnn_small architecture and a differentiable two-block hidden-state Jacobian with tr(K²) regularization. The exact core identity tr(K²)=2ab passed, and the trained-model signature showed a tiny 0.0103% reduction in measured closed-walk gain. However, the task metric did not improve significantly: paired delta=-2.224e-6 with permutation p=0.21825, so this benchmark does not establish a real win.

Baseline: Dynamics/rnn_small; best config lr=0.006, weight_decay=0.0001, epochs=3; test MSE 0.02110967 ± 0.01360108 over 8 seeds.
Idea: Positive-cycle penalty; best config lr=0.006, weight_decay=0.0001, lambda=0.02, tau=1.0, epochs=3; test MSE 0.02110744 ± 0.01359941; paired delta=-0.0000022243; 7/8 wins; permutation p=0.21825; trace2 0.00261571 versus baseline 0.00261598.

Benchmark result