# Tiny Local Recurrence with Adaptive Computation

- ID: 2275
- Canonical URL: https://synthcore.org/idea/2275/tiny-local-recurrence-with-adaptive-computation
- API JSON: https://synthcore.org/api/idea/2275.json
- API Markdown: https://synthcore.org/api/idea/2275.md
- Verification status: beats_baseline
- Source: [arXiv:2608.14019](https://arxiv.org/abs/2608.14019)
- Category: architecture
- Solves: memory, scalability, accuracy
- ML areas: rnn, ssm, mlp, training-dynamics
- Math tags: dynamical-systems, computation, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace a stack of independently parameterized residual or MLP blocks with a small latent grid or vector repeatedly updated by one shared transition rule. Let the number of updates depend on the current latent state, so easy examples terminate early while hard examples receive more computation, potentially improving parameter efficiency and extrapolation.

## Mathematical statement

The paper's central computation is the latent dynamical system $x\to E(x)\to s_0\xrightarrow{f^T}s_T\to D(s_T)\to y$, where $x$ is the external input, $E$ is an encoder, $s_0$ is the latent state, $f$ is one fixed transition map reused at every microstep, $T(s_0)$ is an input-dependent number of iterations, $D$ is the decoder, and $y$ is the output. The trainable supervised objective is $\theta^\star\in\arg\min_\theta\mathbb{E}[\ell(\hat y,y)]$ with $\hat y=\Phi_\theta(x)$. Implement $s_{t+1}=f_\theta(s_t)$ using a local convolutional rule or shared low-rank MLP, and estimate a halting probability $h_t=\sigma(w_h^\top\operatorname{pool}(s_t)+b_h)$; the expected computation is $T$ obtained by accumulating halting mass. The paper's latent-universality claim says that, for some substrates, holding $f$ and the interface fixed while varying the initial condition can realize any partial computable function. This does not prove trainability, but motivates placing task information in $E(x)$ and keeping the recurrent rule extremely small and reusable.

## Key formulas

- $$x\;\to\;E(x)\;\to\;s_{0}\;\xrightarrow{\,f^{T}\,}\;s_{T}\;\to\;D(s_{T})\;\to\;y$$
- $$\theta^{\star}\in\arg\min_{\theta}\mathbb{E}[\ell(\hat{y},y)],\qquad \hat{y}=\Phi_{\theta}(x)$$
- $$s_{t+1}=f_{\theta}(s_t),\qquad h_t=\sigma\!\left(w_h^\top\operatorname{pool}(s_t)+b_h\right),\qquad T(x)=\min\left\{t:\sum_{k=1}^{t}h_k\ge 1-\varepsilon\right\}$$
- $$\mathcal L=\mathbb E[\ell(D(s_T),y)]+\lambda\,\mathbb E[T(x)]$$

## Implementation notes

Integrate this as a replacement for a conventional stack of Transformer feed-forward blocks or residual blocks, leaving the surrounding embedding and output head unchanged. Use a latent tensor $s\in\mathbb R^{C\times H\times W}$ for the simplest experiment, or $s\in\mathbb R^d$ for a vector version. The encoder writes the token or example representation into designated channels or positions, and the decoder reads the final state. Define one shared transition as `s = s + alpha * local_rule(norm(s))`, where `local_rule` is a depthwise 3x3 convolution followed by a pointwise two-layer MLP; all weights are reused at every microstep. Pseudocode: `s=E(x); mass=0; state_acc=0; for t in range(Tmax): s=f(s); h=sigmoid(wh@pool(s)+bh); delta=min(h,1-mass); state_acc += delta*s; mass += delta; if mass >= 1-eps: break; y=D(state_acc + eps*s)`. Compute the task loss plus the ponder penalty $\lambda T$ shown above. The paper contributes the fixed-rule recurrence, interface separation, and adaptive $T$ interpretation; the halting probability and penalty are an explicit engineering adaptation. First test on MNIST, CIFAR-10, and a synthetic arithmetic extrapolation task using a 6-layer residual MLP baseline with matched parameter count. Measure validation accuracy, average microsteps, wall-clock time, FLOPs, and extrapolation accuracy outside the training range. Success means equal accuracy with at least 2x fewer learned parameters, or lower average computation on easy examples without a worse hard-example error curve.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: helps
- Verdict: Built a shared adaptive recurrence classifier and a six-block residual MLP baseline with fixed seeds and CUDA fallback. The toy checks confirmed the predicted stability boundary, exact halting-time formula, and monotonic reduction in halting time as the bias increases. The idea matched baseline accuracy while using 74.2% fewer parameters and 31.5% fewer average microsteps, but it was slower wall-clock because of recurrent Python/control-flow overhead.

### Mechanism check

- Verdict: Built a shared adaptive recurrence classifier and a six-block residual MLP baseline with fixed seeds and CUDA fallback. The toy checks confirmed the predicted stability boundary, exact halting-time formula, and monotonic reduction in halting time as the bias increases. The idea matched baseline accuracy while using 74.2% fewer parameters and 31.5% fewer average microsteps, but it was slower wall-clock because of recurrent Python/control-flow overhead.
- Confidence: 8/10
- Limitations: Only a small 2D synthetic classification task was tested. MNIST, CIFAR-10, arithmetic extrapolation, explicit FLOP measurements, hard/easy error curves, and larger-scale generalization were not tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/907/bench_experiment.py)
- [bench_experiment_corrected.py](https://synthcore.org/code/907/bench_experiment_corrected.py)
- [bench_report.json](https://synthcore.org/code/907/bench_report.json)
- [bench_report_corrected.json](https://synthcore.org/code/907/bench_report_corrected.json)
- [experiment.py](https://synthcore.org/code/907/experiment.py)
- [report.md](https://synthcore.org/code/907/report.md)
- [report_bench_2026-09-02T231432.md](https://synthcore.org/code/907/report_bench_2026-09-02T231432.md)
- [results.json](https://synthcore.org/code/907/results.json)
- [stage2_bench.py](https://synthcore.org/code/907/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/907)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.