# Branch-Free Double-Word FMA Accumulator

- ID: 744
- Canonical URL: https://synthcore.org/idea/744/branch-free-double-word-fma-accumulator
- API JSON: https://synthcore.org/api/idea/744.json
- API Markdown: https://synthcore.org/api/idea/744.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.11391](https://arxiv.org/abs/2607.11391)
- Category: training
- Solves: accuracy, stability, speedup
- ML areas: attention, transformer, optimizer, inference-speedup
- Math tags: numerical-analysis, linear-algebra, approximation-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace ordinary low-precision multiply-add accumulation in selected neural-network reductions with a two-word floating-point accumulator updated by the paper's branch-free DW-FMA network. The high word retains the main sum and the low word stores the rounding residual, improving cancellation behavior without the control-flow divergence of conditional compensated summation.

## Mathematical statement

Represent a double-word number as c=c_0+c_1 with |c_1|\leq u|c_0|, where u=2^{-p} is the unit roundoff of the base binary floating-point format. For x=x_0+x_1 and y=y_0+y_1, TwoProd(a,b) returns p=\mathrm{fl}(ab) and e such that p+e=ab exactly and |e|\leq u|ab|. TwoSum(a,b) returns s,t such that s+t=a+b exactly and |t|\leq u|s|. The branch-free DW-FMA network computes P_{00},E_{00}=    extsc{TwoProd}(x_0,y_0), s,t=    extsc{TwoSum}(P_{00},c_0), \ell=\mathrm{fl}(P_{01}+P_{10}), v=\mathrm{fl}(E_{00}+c_1), w=\mathrm{fl}(v+\ell), t_p=\mathrm{fl}(t+w), and z=    extsc{FastTwoSum}(s,t_p). Here P_{01}=x_0y_1 and P_{10}=x_1y_0. FastTwoSum is valid when exponent(s)\geq exponent(t_p), and returns a normalized non-overlapping pair z=(z_0,z_1). More generally, the EFT sweeps preserve the represented multiset sum, so truncating a K-word expansion gives z-(x\cdot y+c)=-\sum_{i\geq K}v_i and, ideally, |z-(x\cdot y+c)|\leq C_Ku^K|x\cdot y+c|. If residuals overlap, use TwoSum rather than FastTwoSum for safe normalization.

## Key formulas

- $$\textsc{TwoProd}(a,b)=(p,e),\qquad p=\mathrm{fl}(ab),\qquad p+e=ab,\qquad |e|\leq u|ab|.$$
- $$\textsc{TwoSum}(a,b)=(s,t),\qquad s+t=a+b\ \text{exactly},\qquad |t|\leq u|s|.$$
- $$P_{00},E_{00}=\textsc{TwoProd}(x_0,y_0),\quad s,t=\textsc{TwoSum}(P_{00},c_0),\quad \ell=\mathrm{fl}(P_{01}+P_{10}),$$ $$v=\mathrm{fl}(E_{00}+c_1),\quad w=\mathrm{fl}(v+\ell),\quad t_p=\mathrm{fl}(t+w),\quad z=\textsc{FastTwoSum}(s,t_p).$$
- $$z-(x\cdot y+c)=-\sum_{i\geq K}v_i,\qquad |z-(x\cdot y+c)|\leq C_Ku^K|x\cdot y+c|.$$

## Implementation notes

1. Integration point: implement a CUDA, Triton, or custom C++ kernel for selected reductions instead of replacing every tensor operation. The first target should be the inner accumulation of an FP16/BF16 matrix multiplication, an attention-query/key dot product, or an optimizer update. Store each accumulator as two FP32 values, hi and lo. Inputs can be ordinary FP16/BF16 values converted to FP32 with zero low word, or two-word values with nonzero x1 and y1. 2. Pseudocode: initialize hi=0 and lo=0; for each reduction term, set x0=float32(x[k]), x1=0, y0=float32(y[k]), y1=0, c0=hi, c1=lo; compute P00,E00=TwoProd(x0,y0), s,t=TwoSum(P00,c0), ell=fl(P01+P10), v=fl(E00+c1), w=fl(v+ell), tp=fl(t+w), then hi,lo=FastTwoSum(s,tp). Return hi+lo. Implement TwoProd with p=fl(a*b) and e=fma(a,b,-p), and use fixed-operation branch-free TwoSum. 3. The paper's identities are exact; estimate empirically whether |lo|<=u|hi| by logging |lo|/(u|hi|). If this premise fails often, replace the final FastTwoSum with safe TwoSum, accepting extra instructions. 4. First experiment: train a small 2-layer Transformer on WikiText-2 using BF16 activations and compare ordinary FP32 accumulation, Kahan compensation, and DW-FMA at matched batch size and approximate FLOPs. Measure perplexity, gradient variance, loss spikes, NaN/Inf frequency, and kernel throughput. Success means lower perplexity or fewer numerical failures at comparable memory and a smaller slowdown than branch-based compensation; also test long-context attention against an FP64 reference.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built and numerically verified a float32 DW-FMA accumulator with TwoProd, TwoSum, FastTwoSum, safe normalization, length sweeps, controlled cancellation, and full two-word inputs. EFT reconstruction was exact in the tested precision; DW-FMA reduced long-reduction error by about 20x versus naive accumulation and exactly preserved residuals at the predicted float32 cancellation boundary. The mechanism manifested, but this MVP was much slower than naive and Kahan in Python, so no speedup claim is supported.

### Mechanism check

- Verdict: Built and numerically verified a float32 DW-FMA accumulator with TwoProd, TwoSum, FastTwoSum, safe normalization, length sweeps, controlled cancellation, and full two-word inputs. EFT reconstruction was exact in the tested precision; DW-FMA reduced long-reduction error by about 20x versus naive accumulation and exactly preserved residuals at the predicted float32 cancellation boundary. The mechanism manifested, but this MVP was much slower than naive and Kahan in Python, so no speedup claim is supported.
- Confidence: 8/10
- Limitations: No CUDA, Triton, custom compiled kernel, neural-network training, hardware BF16 reduction, FLOP-matched throughput, gradient variance, perplexity, or long-context attention test was performed. TwoProd residuals were simulated with float64 subtraction rather than an actual hardware FMA, and the implementation is a Python reference accumulator.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.56775
- Paired wins: 4/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/311/bench_report.json)
- [dw_fma_bench.py](https://synthcore.org/code/311/dw_fma_bench.py)
- [dw_fma_experiment.py](https://synthcore.org/code/311/dw_fma_experiment.py)
- [report.md](https://synthcore.org/code/311/report.md)
- [report_bench_2026-08-31T010157.md](https://synthcore.org/code/311/report_bench_2026-08-31T010157.md)
- [results.json](https://synthcore.org/code/311/results.json)
- [Download all files as ZIP](https://synthcore.org/download/311)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.