Unverified 2026

Residual-Update Halting

Implementation & benchmark of arXiv:2607.10681 — LayerNorm as Implicit Gain Control in Looped Transformers

Usefulness6/10
Difficulty5/10
Novelty6/10

Source paper: LayerNorm as Implicit Gain Control in Looped Transformers arXiv:2607.10681 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace activation-magnitude-based adaptive computation halting with a criterion based on the actual recurrent update and a local stability margin. The loop halts when the state change is small relative to state scale for several consecutive steps, avoiding pathological decisions when LayerNorm-driven dynamics cause the activation norm to collapse.

Formulas

$$\delta_t=h_{t+1}-h_t=(\bar{A}-\mathbf{1})\odot h_t+f(h_t,e)$$
$$q_t=\frac{\|\delta_t\|_2}{\|h_t\|_2+\epsilon}$$
$$\text{halt at }t\text{ if }q_{t-j}\leq\tau\ \forall j\in\{0,\ldots,m-1\}\quad\text{and}\quad\hat\rho_t\leq r<1$$
$$J_t=\operatorname{Diag}(\bar{A})+\frac{\partial f(h_t,e)}{\partial h_t},\qquad \hat\rho_t\approx\max_i|\lambda_i(J_t)|$$

Mathematical statement

The loop state obeys $h_{t+1}=\bar{A}\odot h_t+f(h_t,e)$. Its actual update is $\delta_t=h_{t+1}-h_t=(\bar{A}-\mathbf{1})\odot h_t+f(h_t,e)$, where $\mathbf{1}$ is the all-ones vector. A fixed point $h^*$ satisfies $\delta_t=0$, so update size is a direct convergence signal, unlike $\|h_t\|$, which can be small even when the recurrence is not settled. Define $q_t=\|\delta_t\|_2/(\|h_t\|_2+\epsilon)$, with $\epsilon>0$. Halt only if $q_t\leq\tau$ for $m$ consecutive iterations and a local stability estimate satisfies $\hat\rho_t\leq r<1$. The local Jacobian is $J_t=\operatorname{Diag}(\bar{A})+\partial f(h_t,e)/\partial h_t$, and $\hat\rho_t$ can be estimated with a few Arnoldi iterations.

Implementation notes

Use this as an inference-time controller for the bounded increment loop, with an optional compute penalty. Run the recurrent update normally, but after computing h_next save delta=h_next-h and q=norm(delta)/(norm(h)+eps). Maintain a counter stable_count; increment it when q<=tau and reset it otherwise. Once stable_count reaches m, estimate the local spectral margin every few iterations rather than every iteration. Obtain the estimate with one or two Arnoldi steps based on Jacobian-vector products through the recurrent update; for a cheaper first version, use the norm ratio ||Jv||/||v|| as a conservative proxy but report it separately because it estimates operator amplification rather than spectral radius. Halt when both conditions hold, or force termination at T_max. Use tau in {1e-2, 3e-3, 1e-3}, m=2 or 3, r=0.98, and eps=1e-6. During training, sample a random stopping time or use the maximum loop count so all examples receive gradients; at inference, optionally add a compute penalty lambda_T*T_used. Compare against ACT based on a learned halting head and against fixed T on delayed-copy, reasoning, and small language-model validation sets. Measure average loop count, accuracy or perplexity at equal compute, premature-halt rate, and correlation between the stopping decision and the remaining-state error ||h_T-h_{T_max}||. Success is at least 25% fewer loop evaluations with no more than 0.5% accuracy loss and substantially fewer failures caused by activation-norm collapse.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.