# Positive-cycle Jacobian penalty

- ID: 1102
- Canonical URL: https://synthcore.org/idea/1102/positive-cycle-jacobian-penalty
- API JSON: https://synthcore.org/api/idea/1102.json
- API Markdown: https://synthcore.org/api/idea/1102.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.18128](https://arxiv.org/abs/2607.18128)
- Category: regularization
- Solves: stability, generalization, accuracy
- ML areas: attention, transformer, moe-routing, regularization
- Math tags: dynamical-systems, spectral-theory, linear-algebra, optimization
- Ratings: usefulness 7/10; difficulty 7/10; novelty 7/10

## Idea description

Penalize short positive feedback cycles in an iterative neural module by suppressing products of absolute Jacobian blocks around the cycle. This targets the mechanism responsible for exponential temperature sensitivity rather than merely penalizing the total Jacobian norm, allowing strong feed-forward paths while controlling recurrent amplification.

## Mathematical statement

The paper distinguishes a directed acyclic feedback graph, whose longest path $L$ yields susceptibility $\Theta(\tau^{-(L+1)})$, from a positive cycle, which produces an $e^{C/\tau}$ factor along a Perron direction. For a nonnegative feedback matrix $K$, a directed cycle $C=(i_1,\ldots,i_r,i_1)$ has gain $G_C=\prod_{a=1}^{r}K_{i_{a+1},i_a}$, where $i_{r+1}=i_1$. Its geometric mean $G_C^{1/r}$ measures the per-edge cyclic amplification. For a nonnegative matrix, $\operatorname{tr}(K^r)$ sums products over all closed walks of length $r$, so it is a differentiable surrogate for short positive cycles. In a neural module partitioned into state blocks $h^1,\ldots,h^m$, define $J_{ij}=|\partial h^i_{new}/\partial h^j_{old}|$. The proposed regularizer controls temperature-scaled closed-walk gain through $\operatorname{tr}((J/\tau)^r)$ rather than penalizing every path equally.

## Key formulas

- $$\text{DAG susceptibility}=\Theta(\tau^{-(L+1)}),\qquad \text{positive cycle susceptibility}\sim e^{C/\tau}$$
- $$G_C=\prod_{a=1}^{r}K_{i_{a+1},i_a},\qquad i_{r+1}=i_1$$
- $$\operatorname{tr}(K^r)=\sum_{i_1,\ldots,i_r}K_{i_1i_r}K_{i_ri_{r-1}}\cdots K_{i_2i_1}$$
- $$\mathcal{L}_{cycle}=\lambda\sum_{r=2}^{R}\frac{\operatorname{tr}(J^r)}{\tau^r m},\qquad J_{ij}=\left|\frac{\partial h^{i}_{new}}{\partial h^{j}_{old}}\right|$$

## Implementation notes

Apply the penalty to a recurrent attention cell, deep equilibrium layer, iterative MoE router, or residual block with explicitly separated state groups. Let $h=(h^1,\ldots,h^m)$ denote block states and estimate the block Jacobian $J_{ij}=|\partial h^i_{new}/\partial h^j_{old}|$ using Hutchinson-style Jacobian products rather than materializing the full matrix. For each minibatch, sample a nonnegative random vector $q$ and compute $u_1=Jq$, $u_2=Ju_1$, up to $u_R=Ju_{R-1}$ with absolute Jacobian-vector products; estimate closed-walk contributions with independent Hutchinson vectors, using $\operatorname{tr}(J^r)\approx q^\top J^r q$ after appropriate normalization. Add $\mathcal{L}_{cycle}=\lambda\sum_{r=2}^{R}\operatorname{tr}(J^r)/(\tau^r m)$ to the task loss, using $R=2$ or $3$ initially. If second-order autodiff is too expensive, compute the Jacobian estimate on detached activations and update the penalty intermittently, or use a finite-difference stop-gradient approximation. For attention, blocks can be heads or token groups; for MoE, they can be experts and router state. The paper supplies the causal-versus-cycle distinction and temperature scaling; the block partition and stochastic trace estimator are engineering choices. Begin with a 4-layer recurrent transformer on CIFAR-10 sequences or WikiText-2, comparing no regularizer, spectral-norm regularization, and this cycle penalty at matched auxiliary-gradient cost. Measure validation loss, gradient spikes, seed variance, effective depth, estimated closed-walk gains, and throughput. The prediction is that cycle regularization reduces instability and seed variance more than a global Jacobian norm penalty at similar task accuracy, while preserving stronger non-cyclic feed-forward interactions.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built and numerically verified a positive-cycle Jacobian penalty MVP for a two-state recurrent cell. The mechanism manifested: tr(K²)=2ab exactly, the growth transition occurred at the predicted gamma*=1/rho=0.833 and was bracketed between 0.76 and 0.90, measured log-growth slopes matched theory exactly, and temperature scaling followed the predicted tau^-2 law. In the mini-experiment, the cycle penalty reduced mean closed-walk gain to 0.0396 versus 0.6008 unregularized and 0.0956 for the spectral proxy, but it did not improve mean task MSE and required about 1.5x the runtime.

### Mechanism check

- Verdict: Built and numerically verified a positive-cycle Jacobian penalty MVP for a two-state recurrent cell. The mechanism manifested: tr(K²)=2ab exactly, the growth transition occurred at the predicted gamma*=1/rho=0.833 and was bracketed between 0.76 and 0.90, measured log-growth slopes matched theory exactly, and temperature scaling followed the predicted tau^-2 law. In the mini-experiment, the cycle penalty reduced mean closed-walk gain to 0.0396 versus 0.6008 unregularized and 0.0956 for the spectral proxy, but it did not improve mean task MSE and required about 1.5x the runtime.
- Confidence: 9/10
- Limitations: This is a tiny two-state synthetic regression experiment, not a transformer, attention model, MoE, or deep-equilibrium system. The Jacobian estimate is exact for the toy block structure but uses detached activations, only r=2, three random seeds, and no matched-FLOP larger-scale validation.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.21825
- Paired wins: 7/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/439/bench_report.json)
- [experiment.py](https://synthcore.org/code/439/experiment.py)
- [multiseed.json](https://synthcore.org/code/439/multiseed.json)
- [report.md](https://synthcore.org/code/439/report.md)
- [report_bench_2026-08-31T122059.md](https://synthcore.org/code/439/report_bench_2026-08-31T122059.md)
- [results.json](https://synthcore.org/code/439/results.json)
- [stage2_bench.py](https://synthcore.org/code/439/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/439)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.