arXiv:2607.10681 2026 Dynamics 3 ideas extracted · analyzed Aug 30, 2026
The paper identifies a useful separation between two mechanisms in looped transformers: a bounded diagonal carry controls convergence, while the nonlinear pre-LayerNorm increment supplies most of the computational memory. Its key stability asset is the increment-form recurrence, which avoids embedding an additional identity path inside the learned increment, together with the observation that local Jacobian spectral radius can be below one even when the operator norm exceeds one because of non-normal dynamics. This suggests building recurrent transformer blocks with explicitly bounded carries, monitoring spectral rather than purely norm-based stability, and using residual-update-based halting instead of activation magnitude. The claims are especially attractive for weight-tied depth expansion and adaptive computation, but should be tested against ordinary recurrent residual transformers.
Regularize the local recurrent Jacobian by its spectral radius rather than imposing the overly conservative operator-norm condition $\|J\|_2<1$. This permits useful non-normal updates with transient amplification while explicitly pushing the asymptotic dynamics toward a stable fixed point.
Construct a weight-tied transformer loop in which the recurrent state receives a bounded diagonal carry plus a learned block increment, rather than applying a residual identity inside the learned increment. Parameterize the carry so every channel is strictly below one, allowing many recurrent iterations without the state explosion observed with an unconstrained carry.
Replace activation-magnitude-based adaptive computation halting with a criterion based on the actual recurrent update and a local stability margin. The loop halts when the state change is small relative to state scale for several consecutive steps, avoiding pathological decisions when LayerNorm-driven dynamics cause the activation norm to collapse.