# Deferred-Correction Adaptive-Depth Block

- ID: 116
- Canonical URL: https://synthcore.org/idea/116/deferred-correction-adaptive-depth-block
- API JSON: https://synthcore.org/api/idea/116.json
- API Markdown: https://synthcore.org/api/idea/116.md
- Verification status: unverified
- Source: [arXiv:2608.23919](https://arxiv.org/abs/2608.23919)
- Category: dynamics
- Solves: speedup, stability, scalability
- ML areas: rnn, training-dynamics, inference-speedup
- Math tags: numerical-analysis, dynamical-systems, approximation-theory
- Ratings: usefulness 6/10; difficulty 4/10; novelty 5/10

## Idea description

Replace a fixed stack of expensive residual blocks with a sequence of increasingly accurate correction stages. Each stage starts from the previous representation, applies a cheap low-capacity update, and optionally performs a more expensive correction; inference halts when the normalized change between successive approximations falls below a tolerance. This creates an anytime network whose computation adapts to the difficulty of each input.

## Mathematical statement

The paper starts from a split dynamical system $\frac{d\bm{u}}{dt}=\bm{S}(t,\bm{u})+\bm{N}(t,\bm{u})$, where $\bm{u}(t)\in\mathbb{R}^{Q}$ is the state, $\bm{S}$ is the stiff component, and $\bm{N}$ is the nonstiff component. Its adaptive criterion is $\frac{\left\lVert\underline{\bm{u}}^{M^{(p)},(p)}-\underline{\bm{u}}^{M^{(p-1)},(p-1)}\right\rVert_{2}}{\left\lVert\underline{\bm{u}}^{M^{(p)},(p)}\right\rVert_{2}}\leq\varepsilon$, where $p$ is the current correction/order, $M^{(p)}$ is the number of discretization nodes used at level $p$, $\underline{\bm{u}}^{M^{(p)},(p)}$ is the current approximation, and $\varepsilon$ is a user tolerance. The neural adaptation uses $h^{(p)}$ for the level-$p$ hidden approximation and $I_{p-1\to p}$ for interpolation or prolongation from the cheaper level to the current representation. The transferable property is the a posteriori relative correction estimate: additional computation is conditional on observed convergence rather than imposed at a fixed depth.

## Key formulas

- $$\frac{d\bm{u}}{dt}=\bm{S}(t,\bm{u}(t))+\bm{N}(t,\bm{u}(t)),\qquad \bm{u}(0)=\bm{z}.$$
- $$\frac{\left\lVert\underline{\bm{u}}^{M^{(p)},(p)}-\underline{\bm{u}}^{M^{(p-1)},(p-1)}\right\rVert_{2}}{\left\lVert\underline{\bm{u}}^{M^{(p)},(p)}\right\rVert_{2}}\leq\varepsilon.$$
- $$h^{(p)}=I_{p-1\to p}\!\left(h^{(p-1)}\right)+C_{p,\theta_p}\!\left(I_{p-1\to p}\!\left(h^{(p-1)}\right),x\right),\qquad r_p=\frac{\left\lVert h^{(p)}-I_{p-1\to p}(h^{(p-1)})\right\rVert_2}{\left\lVert h^{(p)}\right\rVert_2+\delta}.$$
- $$\text{halt at level }p\text{ if }r_p\leq\varepsilon;\qquad p\leftarrow p+1\text{ otherwise}.$$

## Implementation notes

Integrate this at inference time as a replacement for a fixed-depth residual stack, preferably first in a small MLP or Transformer feed-forward sublayer. Let $h^{(0)}$ be the output of a cheap base block. Build levels $p=1,\ldots,P$ with increasing capacity, such as hidden widths $[64,128,256]$, increasing low-rank rank, or progressively larger token-mixing kernels. Define $I_{p-1\to p}$ as an explicit projection from the previous width to the next width; initialize it as nearest-width copying plus a learned linear projection. At each level compute $\tilde h=I_{p-1\to p}(h^{(p-1)})$, compute the correction $c=C_{p,\theta_p}(\tilde h,x)$, and set $h^{(p)}=\tilde h+c$. Initialize the correction branches near zero so that successive levels are refinements rather than unrelated networks. Compute $r_p=\|h^{(p)}-\tilde h\|_2/(\|h^{(p)}\|_2+10^{-6})$ per example, detach $r_p$ only for the halting decision, and stop evaluating later levels when $r_p\leq\varepsilon$. During training, use a fixed maximum $P$ with a weighted loss on every level, or use straight-through halting together with a compute penalty $\lambda p$. The paper supplies the relative-change stopping rule; cost and accuracy thresholds must be estimated empirically. Begin with CIFAR-10 and a three-level residual MLP or small ViT, comparing against a fixed-depth model with equal maximum capacity. Measure accuracy at equal average FLOPs, average executed levels, worst-case latency, and correlation between $r_p$ and final prediction error. Success means at least 25% lower average inference FLOPs with no more than 0.2 percentage points accuracy loss, while difficult examples automatically use deeper levels.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.