# Residual-Update Halting

- ID: 702
- Canonical URL: https://synthcore.org/idea/702/residual-update-halting
- API JSON: https://synthcore.org/api/idea/702.json
- API Markdown: https://synthcore.org/api/idea/702.md
- Verification status: unverified
- Source: [arXiv:2607.10681](https://arxiv.org/abs/2607.10681)
- Category: architecture
- Solves: speedup, stability, memory
- ML areas: transformer, inference-speedup, training-dynamics
- Math tags: dynamical-systems, spectral-theory, linear-algebra
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Replace activation-magnitude-based adaptive computation halting with a criterion based on the actual recurrent update and a local stability margin. The loop halts when the state change is small relative to state scale for several consecutive steps, avoiding pathological decisions when LayerNorm-driven dynamics cause the activation norm to collapse.

## Mathematical statement

The loop state obeys $h_{t+1}=\bar{A}\odot h_t+f(h_t,e)$. Its actual update is $\delta_t=h_{t+1}-h_t=(\bar{A}-\mathbf{1})\odot h_t+f(h_t,e)$, where $\mathbf{1}$ is the all-ones vector. A fixed point $h^*$ satisfies $\delta_t=0$, so update size is a direct convergence signal, unlike $\|h_t\|$, which can be small even when the recurrence is not settled. Define $q_t=\|\delta_t\|_2/(\|h_t\|_2+\epsilon)$, with $\epsilon>0$. Halt only if $q_t\leq\tau$ for $m$ consecutive iterations and a local stability estimate satisfies $\hat\rho_t\leq r<1$. The local Jacobian is $J_t=\operatorname{Diag}(\bar{A})+\partial f(h_t,e)/\partial h_t$, and $\hat\rho_t$ can be estimated with a few Arnoldi iterations.

## Key formulas

- $$\delta_t=h_{t+1}-h_t=(\bar{A}-\mathbf{1})\odot h_t+f(h_t,e)$$
- $$q_t=\frac{\|\delta_t\|_2}{\|h_t\|_2+\epsilon}$$
- $$\text{halt at }t\text{ if }q_{t-j}\leq\tau\ \forall j\in\{0,\ldots,m-1\}\quad\text{and}\quad\hat\rho_t\leq r<1$$
- $$J_t=\operatorname{Diag}(\bar{A})+\frac{\partial f(h_t,e)}{\partial h_t},\qquad \hat\rho_t\approx\max_i|\lambda_i(J_t)|$$

## Implementation notes

Use this as an inference-time controller for the bounded increment loop, with an optional compute penalty. Run the recurrent update normally, but after computing h_next save delta=h_next-h and q=norm(delta)/(norm(h)+eps). Maintain a counter stable_count; increment it when q<=tau and reset it otherwise. Once stable_count reaches m, estimate the local spectral margin every few iterations rather than every iteration. Obtain the estimate with one or two Arnoldi steps based on Jacobian-vector products through the recurrent update; for a cheaper first version, use the norm ratio ||Jv||/||v|| as a conservative proxy but report it separately because it estimates operator amplification rather than spectral radius. Halt when both conditions hold, or force termination at T_max. Use tau in {1e-2, 3e-3, 1e-3}, m=2 or 3, r=0.98, and eps=1e-6. During training, sample a random stopping time or use the maximum loop count so all examples receive gradients; at inference, optionally add a compute penalty lambda_T*T_used. Compare against ACT based on a learned halting head and against fixed T on delayed-copy, reasoning, and small language-model validation sets. Measure average loop count, accuracy or perplexity at equal compute, premature-halt rate, and correlation between the stopping decision and the remaining-state error ||h_T-h_{T_max}||. Success is at least 25% fewer loop evaluations with no more than 0.5% accuracy loss and substantially fewer failures caused by activation-norm collapse.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.