# Bounded Increment Loop

- ID: 700
- Canonical URL: https://synthcore.org/idea/700/bounded-increment-loop
- API JSON: https://synthcore.org/api/idea/700.json
- API Markdown: https://synthcore.org/api/idea/700.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.10681](https://arxiv.org/abs/2607.10681)
- Category: architecture
- Solves: stability, scalability, accuracy
- ML areas: transformer, training-dynamics, initialization
- Math tags: dynamical-systems, linear-algebra, stability
- Ratings: usefulness 7/10; difficulty 4/10; novelty 5/10

## Idea description

Construct a weight-tied transformer loop in which the recurrent state receives a bounded diagonal carry plus a learned block increment, rather than applying a residual identity inside the learned increment. Parameterize the carry so every channel is strictly below one, allowing many recurrent iterations without the state explosion observed with an unconstrained carry.

## Mathematical statement

The recurrence is $h_{t+1}=\bar{A}\odot h_t+f(h_t,e)$, where $h_t\in\mathbb{R}^{n\times d}$ is the loop state, $e$ is the fixed encoded input, $\bar{A}\in(0,1)^d$ is broadcast over sequence positions, and $\odot$ is elementwise multiplication. The increment is $f(h_t,e)=\operatorname{Block}(h_t+e)-(h_t+e)$. The carry is parameterized as $\bar{A}=\exp(\Delta t\cdot(-\exp(\log_A)))$, with $\Delta t>0$ and learnable $\log_A\in\mathbb{R}^d$, guaranteeing $0<\bar{A}_c<1$. The structural condition is that $f$ contains no identity term in $h_t$: if $f(h,e)=h+u(h,e)$, the effective linear component becomes $(\bar{A}+\mathbf{1})\odot h_t+u(h,e)$, whose carry entries exceed one. The module therefore uses pre-LayerNorm attention and MLP increments only.

## Key formulas

- $$h_{t+1}=\bar{A}\odot h_t+f(h_t,e),\qquad f(h_t,e)=\operatorname{Block}(h_t+e)-(h_t+e)$$
- $$\bar{A}=\exp\!\left(\Delta t\cdot(-\exp(\log_A))\right),\qquad 0<\bar{A}_c<1$$
- $$h_{t+1}^{\mathrm{bad}}=\bar{A}\odot h_t+h_t+u(h_t,e)=(\bar{A}+\mathbf{1})\odot h_t+u(h_t,e)$$

## Implementation notes

Integrate this at the depth-expansion point of a weight-tied transformer. Let the input encoder produce a fixed tensor e of shape [batch, sequence, d], and initialize h_0 to zeros or a learned per-channel state. Define one ordinary pre-LayerNorm transformer block B(x), but return only its increment B(x)-x. In every loop iteration compute x=h_t+e, y=B(x), increment=y-x, and update h=(A*h)+increment, where A is a [d] vector broadcast over batch and sequence. Store an unconstrained parameter log_A and compute A=exp(-dt*exp(log_A)); clamp log_A to a practical range such as [-8, 8] only for numerical safety. Do not add another residual h inside the increment path. Pseudocode: e=Encoder(input); h=zeros_like(e); A=exp(-dt*exp(log_A)); for t in 1..T: x=h+e; increment=Block(x)-x; h=A*h+increment; output=Readout(h). Compute the paper's parameterization exactly and estimate nothing mathematically. Compare against a tied transformer with an unconstrained learned carry and against a standard recurrent residual block at equal block evaluations. Start with a 6-layer-equivalent loop on CIFAR-10, copy-task sequences, and a small language-model benchmark. Record state norms, gradient norms, validation loss, and maximum A. Success means no state explosion at learning rates where the unconstrained baseline diverges, plus equal-or-better loss at fixed FLOPs. A second experiment should increase T from 4 to 32 and test whether the bounded increment loop preserves accuracy without requiring untied parameters.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built the bounded increment recurrence with the exact A=exp(-dt*exp(log_A)) parameterization, plus deterministic toy verification and a tied-loop regression comparison. The predicted decay law A^T and bad/good growth ratio ((A+1)/A)^T matched observed sweeps with maximum relative errors 4.39e-16 and 1.14e-15; the q sweep showed the predicted stability boundary at q=1. In training, the bounded loop remained stable through T=32 with maximum state RMS about 0.73, while the extra-identity residual variant reached 4.67e8 state RMS and stopped after 5 steps; however, the unconstrained-carry baseline had lower final loss, 0.121 versus 0.142 for bounded, so no overall accuracy win was demonstrated.

### Mechanism check

- Verdict: Built the bounded increment recurrence with the exact A=exp(-dt*exp(log_A)) parameterization, plus deterministic toy verification and a tied-loop regression comparison. The predicted decay law A^T and bad/good growth ratio ((A+1)/A)^T matched observed sweeps with maximum relative errors 4.39e-16 and 1.14e-15; the q sweep showed the predicted stability boundary at q=1. In training, the bounded loop remained stable through T=32 with maximum state RMS about 0.73, while the extra-identity residual variant reached 4.67e8 state RMS and stopped after 5 steps; however, the unconstrained-carry baseline had lower final loss, 0.121 versus 0.142 for bounded, so no overall accuracy win was demonstrated.
- Confidence: 8/10
- Limitations: This is a NumPy toy linear-block regression, not CIFAR-10, language modeling, or a full attention transformer. The unconstrained baseline was only tested from a mildly superunit carry and did not diverge in this setup; no FLOP-normalized large-scale accuracy or multiple-seed study was performed. Float64 underflow occurs for exp(-exp(8)) if the extreme upper clamp is included, although the mathematical value remains strictly positive.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.7094
- Paired wins: 4/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_bounded_loop.py](https://synthcore.org/code/300/bench_bounded_loop.py)
- [bench_report.json](https://synthcore.org/code/300/bench_report.json)
- [bounded_increment_loop.py](https://synthcore.org/code/300/bounded_increment_loop.py)
- [report.md](https://synthcore.org/code/300/report.md)
- [report_bench_2026-08-31T004242.md](https://synthcore.org/code/300/report_bench_2026-08-31T004242.md)
- [results.json](https://synthcore.org/code/300/results.json)
- [Download all files as ZIP](https://synthcore.org/download/300)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.