△ Mechanism confirmed, baseline not beaten 2026

Branch-Free Double-Word FMA Accumulator

Implementation & benchmark of arXiv:2607.11391 — Performance evaluation of branch-free fused multiply-add algorithms for multi-component-type multiple-precision floating-point arithmetic

Usefulness7/10

Difficulty5/10

Novelty6/10

Training Accuracy Stability Speedup Attention Transformer Optimizer Inference speedup Numerical analysis Linear algebra Approximation theory

Source paper: Performance evaluation of branch-free fused multiply-add algorithms for multi-component-type multiple-precision floating-point arithmetic arXiv:2607.11391 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace ordinary low-precision multiply-add accumulation in selected neural-network reductions with a two-word floating-point accumulator updated by the paper's branch-free DW-FMA network. The high word retains the main sum and the low word stores the rounding residual, improving cancellation behavior without the control-flow divergence of conditional compensated summation.

Formulas

$$\textsc{TwoProd}(a,b)=(p,e),\qquad p=\mathrm{fl}(ab),\qquad p+e=ab,\qquad |e|\leq u|ab|.$$

$$\textsc{TwoSum}(a,b)=(s,t),\qquad s+t=a+b\ \text{exactly},\qquad |t|\leq u|s|.$$

$$P_{00},E_{00}=\textsc{TwoProd}(x_0,y_0),\quad s,t=\textsc{TwoSum}(P_{00},c_0),\quad \ell=\mathrm{fl}(P_{01}+P_{10}),$$ $$v=\mathrm{fl}(E_{00}+c_1),\quad w=\mathrm{fl}(v+\ell),\quad t_p=\mathrm{fl}(t+w),\quad z=\textsc{FastTwoSum}(s,t_p).$$

$$z-(x\cdot y+c)=-\sum_{i\geq K}v_i,\qquad |z-(x\cdot y+c)|\leq C_Ku^K|x\cdot y+c|.$$

Mathematical statement

Represent a double-word number as c=c_0+c_1 with |c_1|\leq u|c_0|, where u=2^{-p} is the unit roundoff of the base binary floating-point format. For x=x_0+x_1 and y=y_0+y_1, TwoProd(a,b) returns p=\mathrm{fl}(ab) and e such that p+e=ab exactly and |e|\leq u|ab|. TwoSum(a,b) returns s,t such that s+t=a+b exactly and |t|\leq u|s|. The branch-free DW-FMA network computes P_{00},E_{00}= extsc{TwoProd}(x_0,y_0), s,t= extsc{TwoSum}(P_{00},c_0), \ell=\mathrm{fl}(P_{01}+P_{10}), v=\mathrm{fl}(E_{00}+c_1), w=\mathrm{fl}(v+\ell), t_p=\mathrm{fl}(t+w), and z= extsc{FastTwoSum}(s,t_p). Here P_{01}=x_0y_1 and P_{10}=x_1y_0. FastTwoSum is valid when exponent(s)\geq exponent(t_p), and returns a normalized non-overlapping pair z=(z_0,z_1). More generally, the EFT sweeps preserve the represented multiset sum, so truncating a K-word expansion gives z-(x\cdot y+c)=-\sum_{i\geq K}v_i and, ideally, |z-(x\cdot y+c)|\leq C_Ku^K|x\cdot y+c|. If residuals overlap, use TwoSum rather than FastTwoSum for safe normalization.

Implementation notes

1. Integration point: implement a CUDA, Triton, or custom C++ kernel for selected reductions instead of replacing every tensor operation. The first target should be the inner accumulation of an FP16/BF16 matrix multiplication, an attention-query/key dot product, or an optimizer update. Store each accumulator as two FP32 values, hi and lo. Inputs can be ordinary FP16/BF16 values converted to FP32 with zero low word, or two-word values with nonzero x1 and y1. 2. Pseudocode: initialize hi=0 and lo=0; for each reduction term, set x0=float32(x[k]), x1=0, y0=float32(y[k]), y1=0, c0=hi, c1=lo; compute P00,E00=TwoProd(x0,y0), s,t=TwoSum(P00,c0), ell=fl(P01+P10), v=fl(E00+c1), w=fl(v+ell), tp=fl(t+w), then hi,lo=FastTwoSum(s,tp). Return hi+lo. Implement TwoProd with p=fl(a*b) and e=fma(a,b,-p), and use fixed-operation branch-free TwoSum. 3. The paper's identities are exact; estimate empirically whether |lo|<=u|hi| by logging |lo|/(u|hi|). If this premise fails often, replace the final FastTwoSum with safe TwoSum, accepting extra instructions. 4. First experiment: train a small 2-layer Transformer on WikiText-2 using BF16 activations and compare ordinary FP32 accumulation, Kahan compensation, and DW-FMA at matched batch size and approximate FLOPs. Measure perplexity, gradient variance, loss spikes, NaN/Inf frequency, and kernel throughput. Success means lower perplexity or fewer numerical failures at comparable memory and a smaller slowdown than branch-based compensation; also test long-context attention against an FP64 reference.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built and numerically verified a float32 DW-FMA accumulator with TwoProd, TwoSum, FastTwoSum, safe normalization, length sweeps, controlled cancellation, and full two-word inputs. EFT reconstruction was exact in the tested precision; DW-FMA reduced long-reduction error by about 20x versus naive accumulation and exactly preserved residuals at the predicted float32 cancellation boundary. The mechanism manifested, but this MVP was much slower than naive and Kahan in Python, so no speedup claim is supported.

Agent confidence: 8/10

Baseline: Naive float32 accumulation: median relative error 4.44e-7 at n=8192; controlled cancellation at large=2^24 produced absolute error 1, 4, 16, 64, and 256 for 1, 4, 16, 64, and 256 blocks. Timing for 8192 terms x5: 0.0541 s.
Idea: DW-FMA: median relative error 2.16e-8 at n=8192, maximum |lo|/(u|hi|)=0.999; controlled cancellation at large=2^24 had zero error for all tested block counts. Timing for 8192 terms x5: 1.1087 s. Kahan matched DW-FMA on the random length sweep and had cancellation errors 0, 1, 5, 21, and 85.

Limitations: No CUDA, Triton, custom compiled kernel, neural-network training, hardware BF16 reduction, FLOP-matched throughput, gradient variance, perplexity, or long-context attention test was performed. TwoProd residuals were simulated with float64 subtraction rather than an actual hardware FMA, and the implementation is a Python reference accumulator.

How to run: python3 dw_fma_experiment.py > results.json

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.11391) Sep 2, 2026

Ahead of the baseline (−0.0%) but not statistically significant (p=0.5677, wins 4 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.3541
Idea mean: 0.354
Effect (Δ): -0.0001 (−0.0%; negative = idea better)
Wins: 4 / 8 paired seeds
p-value: 0.5677 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

This is a Python/PyTorch reference implementation, not an optimized fused CUDA, Triton, or C++ kernel, so throughput was not fairly tested. Only the small sequence track and 10 epochs were evaluated; BF16 hardware accumulation, long-context attention, other tracks, gradient variance, and NaN frequency were omitted.

How to run: python3 dw_fma_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Built a matched transformer benchmark on the structurally appropriate sequence track, replacing ordinary attention score reductions with a two-word DW-FMA reduction while keeping the architecture, data, optimizer, epochs, and learning-rate union matched. The trained-model mechanism signature was confirmed, but the task metric showed no significant improvement.

Baseline: Best lr=0.001; 8-seed test MSE mean 0.3540758267, std 0.0626762059.
Idea: Best lr=0.001; 8-seed test MSE mean 0.3540073913, std 0.0625976739; paired delta -0.0000684354, 4/8 wins, permutation p=0.56775. Trained attention-score absolute error was 6.6333178e-7 for ordinary reduction versus 0 for DW-FMA; confirmed=true.

Benchmark result