# Spectral-gap adaptive halting

- ID: 2748
- Canonical URL: https://synthcore.org/idea/2748/spectral-gap-adaptive-halting
- API JSON: https://synthcore.org/api/idea/2748.json
- API Markdown: https://synthcore.org/api/idea/2748.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.26556](https://arxiv.org/abs/2608.26556)
- Category: dynamics
- Solves: inference-speedup, stability
- ML areas: transformer, inference-speedup, training-dynamics
- Math tags: dynamical-systems, bifurcations, linear-algebra
- Ratings: usefulness 8/10; difficulty 5/10; novelty 6/10

## Idea description

Use the local Jacobian of a looped transformer to estimate its remaining relaxation time and stop the recurrent computation when the predicted residual reduction is sufficient. Near a saddle-node fold, the paper's asymptotic relation converts an estimated dominant eigenvalue into a compute forecast, allowing dynamic iteration budgets instead of a conservative fixed maximum.

## Mathematical statement

The recurrent model is the discrete dynamical system \(z_{t+1}=F_{\theta}(z_t;h)\), where \(z_t\) is the hidden state, \(h\) is the input representation, and \(F_\theta\) is one weight-tied transformer block. Let \(J_t=\partial F_\theta(z_t;h)/\partial z_t\) be its local Jacobian and let \(\lambda_{\max}(-\varepsilon)\) denote the dominant eigenvalue in the paper's fold normal-form parameterization, with \(\varepsilon\) measuring distance from the saddle-node transition. The paper states that the relaxation time \(\tau(\varepsilon)\) and spectral gap satisfy \(\tau(\varepsilon)[1-\lambda_{\max}(-\varepsilon)]\to\pi\). We operationalize this by estimating the dominant magnitude eigenvalue \(\hat\lambda_t\) of \(J_t\) with Jacobian-vector products and using \(\hat\tau_t=\pi/(1-\hat\lambda_t)\) as a remaining-iteration estimate. This law is expected only near the fold phase; oscillatory or Neimark-Sacker-like dynamics should be detected by non-monotone residuals and handled by a fallback cap.

## Key formulas

- $$z_{t+1}=F_{\theta}(z_t;h)$$
- $$\tau(\varepsilon)\,[1-\lambda_{\max}(-\varepsilon)]\to\pi$$
- $$J_t=\frac{\partial F_{\theta}(z_t;h)}{\partial z_t},\qquad \hat{\tau}_t=\frac{\pi}{\max(\delta,1-\hat\lambda_t)}$$
- $$\hat\lambda_t\approx\frac{\|J_t v_t\|_2}{\|v_t\|_2},\qquad v_{t+1}=\frac{J_t v_t}{\|J_t v_t\|_2}$$

## Implementation notes

Integrate this into the halting controller of a looped transformer, immediately after each application of the tied block. Keep the usual task output head, but expose the hidden state and a scalar residual such as \(r_t=\|z_t-z_{t-1}\|_2/(\|z_t\|_2+10^{-6})\). At every few iterations, estimate local criticality with one or two power iterations: initialize a random vector \(v\), compute the Jacobian-vector product \(u=J_t v\) using forward-mode autodiff or reverse-mode VJP tricks, set \(\hat\lambda_t=\|u\|_2/\|v\|_2\), and normalize \(v=u/\|u\|_2\). Predict the remaining relaxation time as \(\hat\tau_t=\pi/\max(\delta,1-\hat\lambda_t)\), with \(\delta\) around 0.01 to avoid numerical explosion. Stop when both \(r_t<r_{\mathrm{tol}}\) and the predicted remaining time is below a small threshold; otherwise continue until \(t_{\max}\). Track the last four residuals and disable the fold predictor when they oscillate or increase, using a fixed safe cap for that example. The mathematical quantity is the asymptotic \(\pi\)-calibrated spectral-gap predictor; the empirical choices are the residual threshold, sampling frequency, cap, and oscillation detector. First test on a 4-8 layer weight-tied transformer solving small algorithmic reasoning tasks, comparing fixed-depth inference and standard residual-based early exit at equal accuracy. Success means lower mean iterations and wall-clock cost at the same accuracy, with the predicted \(\hat\tau_t\) correlating with actual iterations and no instability on oscillatory examples.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built a readable MVP for spectral-gap prediction and adaptive halting in the fold normal form. The predicted epsilon and b exponents matched closely (+0.500 for the gap and about -0.489 for passage time versus predicted -0.500), while the calibrated product Pi averaged 3.189 versus pi=3.142, a 1.51% error. The controller did not yield a speedup: fixed and residual methods averaged 449 steps, whereas the strict spectral controller hit its 50,000-step cap, so a practical transformer inference win was not demonstrated.

### Mechanism check

- Verdict: Built a readable MVP for spectral-gap prediction and adaptive halting in the fold normal form. The predicted epsilon and b exponents matched closely (+0.500 for the gap and about -0.489 for passage time versus predicted -0.500), while the calibrated product Pi averaged 3.189 versus pi=3.142, a 1.51% error. The controller did not yield a speedup: fixed and residual methods averaged 449 steps, whereas the strict spectral controller hit its 50,000-step cap, so a practical transformer inference win was not demonstrated.
- Confidence: 9/10
- Limitations: This is a scalar fold-normal-form experiment, not a trained looped transformer. It does not test transformer task accuracy, GPU wall-clock cost, autodiff JVP overhead, learned power-iteration error, or tuned controller thresholds; the practical adaptive-halting benefit remains unverified and was absent in this toy controller.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 1
- Paired wins: 0/8
- Benchmark verdict: no measurable effect

## Artifacts

- [REPORT.md](https://synthcore.org/code/1059/REPORT.md)
- [bench_report.json](https://synthcore.org/code/1059/bench_report.json)
- [experiment.py](https://synthcore.org/code/1059/experiment.py)
- [report.md](https://synthcore.org/code/1059/report.md)
- [report_bench_2026-09-01T232745.md](https://synthcore.org/code/1059/report_bench_2026-09-01T232745.md)
- [results.json](https://synthcore.org/code/1059/results.json)
- [smoke.py](https://synthcore.org/code/1059/smoke.py)
- [spectral_bench.py](https://synthcore.org/code/1059/spectral_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1059)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.