A Second-Moment Theory for Floating-Point Reduction Trees

What the math gives to ML

The paper provides a tree-dependent second-moment model for floating-point reductions, replacing topology-blind worst-case bounds with a common-ancestor kernel and two scalar costs. This is directly relevant to distributed gradient accumulation, tensor-parallel all-reduce, mixture-of-experts routing statistics, and mixed-precision optimizer updates, where identical values can experience materially different rounding error depending on the reduction schedule. The most promising transfer is to choose reduction trees using measured coordinate-wise means and variances, minimizing predicted stochastic error while preserving the communication budget. A small-scale implementation can compare ring, pairwise, blocked, and variance-aware trees under FP16 or BF16 accumulation and measure optimizer stability at equal communication cost.