Idea description

Use the local Jacobian of a looped transformer to estimate its remaining relaxation time and stop the recurrent computation when the predicted residual reduction is sufficient. Near a saddle-node fold, the paper's asymptotic relation converts an estimated dominant eigenvalue into a compute forecast, allowing dynamic iteration budgets instead of a conservative fixed maximum.

Formulas

$$z_{t+1}=F_{\theta}(z_t;h)$$

$$\tau(\varepsilon)\,[1-\lambda_{\max}(-\varepsilon)]\to\pi$$

$$J_t=\frac{\partial F_{\theta}(z_t;h)}{\partial z_t},\qquad \hat{\tau}_t=\frac{\pi}{\max(\delta,1-\hat\lambda_t)}$$

$$\hat\lambda_t\approx\frac{\|J_t v_t\|_2}{\|v_t\|_2},\qquad v_{t+1}=\frac{J_t v_t}{\|J_t v_t\|_2}$$

Mathematical statement

The recurrent model is the discrete dynamical system $z_{t+1}=F_{\theta}(z_t;h)$, where $z_t$ is the hidden state, $h$ is the input representation, and $F_\theta$ is one weight-tied transformer block. Let $J_t=\partial F_\theta(z_t;h)/\partial z_t$ be its local Jacobian and let $\lambda_{\max}(-\varepsilon)$ denote the dominant eigenvalue in the paper's fold normal-form parameterization, with $\varepsilon$ measuring distance from the saddle-node transition. The paper states that the relaxation time $\tau(\varepsilon)$ and spectral gap satisfy $\tau(\varepsilon)[1-\lambda_{\max}(-\varepsilon)]\to\pi$. We operationalize this by estimating the dominant magnitude eigenvalue $\hat\lambda_t$ of $J_t$ with Jacobian-vector products and using $\hat\tau_t=\pi/(1-\hat\lambda_t)$ as a remaining-iteration estimate. This law is expected only near the fold phase; oscillatory or Neimark-Sacker-like dynamics should be detected by non-monotone residuals and handled by a fallback cap.

Implementation notes

Integrate this into the halting controller of a looped transformer, immediately after each application of the tied block. Keep the usual task output head, but expose the hidden state and a scalar residual such as $r_t=\|z_t-z_{t-1}\|_2/(\|z_t\|_2+10^{-6})$. At every few iterations, estimate local criticality with one or two power iterations: initialize a random vector $v$, compute the Jacobian-vector product $u=J_t v$ using forward-mode autodiff or reverse-mode VJP tricks, set $\hat\lambda_t=\|u\|_2/\|v\|_2$, and normalize $v=u/\|u\|_2$. Predict the remaining relaxation time as $\hat\tau_t=\pi/\max(\delta,1-\hat\lambda_t)$, with $\delta$ around 0.01 to avoid numerical explosion. Stop when both $r_t<r_{\mathrm{tol}}$ and the predicted remaining time is below a small threshold; otherwise continue until $t_{\max}$. Track the last four residuals and disable the fold predictor when they oscillate or increase, using a fixed safe cap for that example. The mathematical quantity is the asymptotic $\pi$-calibrated spectral-gap predictor; the empirical choices are the residual threshold, sampling frequency, cap, and oscillation detector. First test on a 4-8 layer weight-tied transformer solving small algorithmic reasoning tasks, comparing fixed-depth inference and standard residual-based early exit at equal accuracy. Success means lower mean iterations and wall-clock cost at the same accuracy, with the predicted $\hat\tau_t$ correlating with actual iterations and no instability on oscillatory examples.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a readable MVP for spectral-gap prediction and adaptive halting in the fold normal form. The predicted epsilon and b exponents matched closely (+0.500 for the gap and about -0.489 for passage time versus predicted -0.500), while the calibrated product Pi averaged 3.189 versus pi=3.142, a 1.51% error. The controller did not yield a speedup: fixed and residual methods averaged 449 steps, whereas the strict spectral controller hit its 50,000-step cap, so a practical transformer inference win was not demonstrated.

Agent confidence: 9/10

Baseline: Fixed/residual stable-branch halting: mean 449 steps over 20 tasks.
Idea: Spectral controller: mean 50,000 steps; no speedup. Mechanism checks: gap-epsilon exponent 0.500, passage-epsilon exponent -0.489, gap-b exponent 0.500, passage-b exponent -0.490, Pi mean 3.189 with 1.51% relative error.

Limitations: This is a scalar fold-normal-form experiment, not a trained looped transformer. It does not test transformer task accuracy, GPU wall-clock cost, autodiff JVP overhead, learned power-iteration error, or tuned controller thresholds; the practical adaptive-halting benefit remains unverified and was absent in this toy controller.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.26556) Sep 2, 2026

No significant difference vs. the tuned baseline (0 metric points, p=1, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0
Idea mean: 0.0043
Effect (Δ): 0 (n/a; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 1 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.0%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 0 configs

Limitations:

This tested the intervention on the built-in GRU dynamics model rather than a looped transformer. Wall-clock/JVP overhead was not separately benchmarked, the controller used a finite-difference directional JVP rather than an exact autodiff power iteration, and the adaptive rule did not reduce the selected cap on this task. The stage-2 result is therefore a no-effect outcome, not evidence of an inference-speed win.

How to run: python3 spectral_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Built a tied-GRU refinement system on the structurally matched dynamics track, comparing fixed-cap inference against spectral-gap adaptive halting with identical independently trained architectures, data, epochs, and shared hyperparameter settings. Across 8 paired seeds, the best idea and baseline had identical MSE (0.0043454), paired delta 0 and permutation p=1.0; the mechanism signature was also not confirmed because predicted tau did not correlate with observed iterations.

Baseline: Best config lr=0.01, cap=4; full 8-seed mean test MSE=0.004345412977272645.
Idea: Best config lr=0.01, cap=4, rtol=0.025, tau=3.0; full 8-seed mean test MSE=0.004345412977272645; paired delta=0.0, p=1.0; mean predicted tau=7.15865, mean observed iterations=4.0, correlation=0.0, confirmed=false.

Benchmark result