# Lattice Error-Feedback Residual Blocks

- ID: 1318
- Canonical URL: https://synthcore.org/idea/1318/lattice-error-feedback-residual-blocks
- API JSON: https://synthcore.org/api/idea/1318.json
- API Markdown: https://synthcore.org/api/idea/1318.md
- Verification status: beats_baseline
- Source: [arXiv:2607.23390](https://arxiv.org/abs/2607.23390)
- Category: architecture
- Solves: accuracy, stability, scalability
- ML areas: quantization, cnn, transformer, inference-speedup
- Math tags: numerical-analysis, dynamical-systems, linear-algebra, approximation-theory
- Ratings: usefulness 8/10; difficulty 5/10; novelty 6/10

## Idea description

Replace full-state quantized write-back in a deep low-bit residual stack with quantized increment error feedback. The residual branch quantizes the proposed increment after adding the previous carry, while the carry stores the exact discrepancy; this makes the total error telescope instead of accumulating approximately once per layer.

## Mathematical statement

The paper distinguishes full-state write-back, whose execution error can incur a depth-dependent penalty of the form D rho_z, from increment error feedback, which has a bounded carry term and an exact common-lattice conservation law. Let z_t be the hidden state, Delta_t be the real-valued residual increment proposed by block t, Q_h be a componentwise quantizer onto lattice h Z^d, and c_t be the stored quantization carry. Use c_0=0, q_t=Q_h(Delta_t+c_t), z_{t+1}=z_t+q_t, and c_{t+1}=Delta_t+c_t-q_t. The conservation identity is c_{t+1}=c_t+Delta_t-q_t, hence sum_{t=0}^{D-1} q_t=sum_{t=0}^{D-1} Delta_t-c_D and z_D=z_0+sum_t Delta_t-c_D. If the carry is not saturated, each coordinate satisfies |c_{D,i}|<h/2 for nearest rounding, so the discrepancy from the unquantized residual trajectory is bounded by ||c_D|| rather than D times the per-step rounding radius.

## Key formulas

- $$q_t=Q_h(\Delta_t+c_t),\qquad z_{t+1}=z_t+q_t,\qquad c_{t+1}=\Delta_t+c_t-q_t,\qquad c_0=0.$$
- $$\sum_{t=0}^{D-1}q_t=\sum_{t=0}^{D-1}\Delta_t-c_D,\qquad z_D=z_0+\sum_{t=0}^{D-1}\Delta_t-c_D.$$
- $$|c_{D,i}|<\frac{h}{2}\ \text{for unsaturated nearest-lattice rounding},\qquad \left|E^{\mathrm{impl}}_{D,s}-E_{\Omega,\infty}\right|\leq\frac{C_{\mathrm{syn}}}{D}+C_{\mathrm{meta}}2^{-s/m}+\mathcal{A}_{D}.$$

## Implementation notes

Integrate this into the residual addition of a post-training or quantization-aware-training ResNet, Transformer residual stream, or MLP residual stack. Keep the main hidden state z in an int8- or int4-compatible fixed-point format and maintain one higher-precision carry tensor c per residual stream; fp16 carry is sufficient for the first experiment. For block t, compute the ordinary floating-point residual proposal Delta=alpha_t*F_t(z), add the carry u=Delta+c, quantize u with a fixed symmetric scale h_t, write q=clip(round(u/h_t))*h_t, update z=z+q, and update c=u-q. During training, backpropagate through round with a straight-through estimator or simulated additive noise; at inference, use exact integer or fixed-point rounding. Pseudocode is: `c=zeros; z=input; for t in range(D): Delta=alpha[t]*F[t](z); u=Delta+c; q=clip(round(u/h[t])*h[t]); z=z+q; c=u-q`. Track maximum and RMS carry, saturation count, final output error, and the ratio of error to depth. The telescoping identity is computed exactly; the unsaturated carry range and structural residual error are estimated empirically. First test CIFAR-10 with a 20-block ResNet using int4 residual increments, comparing ordinary int4 state write-back, stochastic rounding, and this method at equal scales and FLOPs. Success is a flatter output-error-versus-depth curve, at least 2x lower validation error at fixed bit width, or equivalent accuracy with fewer bits. Failure is frequent carry saturation or no improvement over stochastic rounding.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: helps
- Verdict: Built and numerically verified scalar lattice error-feedback residual updates. The conservation identity held to 4.4e-16, baseline full-state error grew with depth at 0.49h per layer as predicted, while feedback error stayed below h/2; carry magnitude scaled linearly with h and remained 0.5h normalized. The mechanism clearly manifested, although this does not establish end-to-end neural-network accuracy or speed gains.

### Mechanism check

- Verdict: Built and numerically verified scalar lattice error-feedback residual updates. The conservation identity held to 4.4e-16, baseline full-state error grew with depth at 0.49h per layer as predicted, while feedback error stayed below h/2; carry magnitude scaled linearly with h and remained 0.5h normalized. The mechanism clearly manifested, although this does not establish end-to-end neural-network accuracy or speed gains.
- Confidence: 9/10
- Limitations: Only a scalar toy residual stream was tested, with fixed nearest rounding and no learned CNN, ResNet, Transformer, CIFAR data, clipping stress test, training dynamics, GPU execution, memory accounting, or wall-clock speed measurement. The bounded-carry result assumes no saturation and does not test finite-range int4/int8 state interactions.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [README.md](https://synthcore.org/code/355/README.md)
- [bench_report.json](https://synthcore.org/code/355/bench_report.json)
- [lattice_bench.py](https://synthcore.org/code/355/lattice_bench.py)
- [official_bench.py](https://synthcore.org/code/355/official_bench.py)
- [report.md](https://synthcore.org/code/355/report.md)
- [report_bench_2026-08-31T092558.md](https://synthcore.org/code/355/report_bench_2026-08-31T092558.md)
- [results.json](https://synthcore.org/code/355/results.json)
- [toy_experiment.py](https://synthcore.org/code/355/toy_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/355)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.