# Tensor-Core Limb Expansion for Stable Accumulation

- ID: 501
- Canonical URL: https://synthcore.org/idea/501/tensor-core-limb-expansion-for-stable-accumulation
- API JSON: https://synthcore.org/api/idea/501.json
- API Markdown: https://synthcore.org/api/idea/501.md
- Verification status: unverified
- Source: [arXiv:2607.06881](https://arxiv.org/abs/2607.06881)
- Category: training
- Solves: stability, accuracy
- ML areas: optimizer, transformer, training, inference-speedup
- Math tags: numerical-analysis, linear-algebra, tensor-decomposition
- Ratings: usefulness 5/10; difficulty 6/10; novelty 6/10

## Idea description

Represent selected activations, weights, or optimizer accumulators as four floating-point limbs and evaluate products through tensor-core matrix multiplications encoding limb convolution. Retain the convolution components during reductions and renormalize only at block boundaries, avoiding branch-heavy multi-double arithmetic inside every multiply-add.

## Mathematical statement

The paper represents a multiple-double scalar by four components \(a_{i,k}=(a_{i,k,0},a_{i,k,1},a_{i,k,2},a_{i,k,3})\), with represented value \(a^{\mathrm{MD}}_{i,k}=\sum_{r=0}^{3}a_{i,k,r}\), and similarly \(b^{\mathrm{MD}}_{k,j}=\sum_{s=0}^{3}b_{k,j,s}\). The product component of output index \(q\) is the convolution \(c_{i,j,q}=\sum_{r+s=q}a_{i,k,r}b_{k,j,s}\), truncated to \(q\in\{0,1,2,3\}\). Equation (2) encodes this convolution with \(A_{i,k}=[a_{i,k,0},a_{i,k,1},a_{i,k,2},a_{i,k,3}]^T\) and an upper-triangular Toeplitz matrix \(B_{k,j}\) whose diagonals contain the \(b\)-limbs. For a neural matrix product, write \(W=\sum_{r=0}^{3}W_r\) and \(X=\sum_{s=0}^{3}X_s\), compute \(Y_q=\sum_{r+s=q}W_rX_s\), and reconstruct \(Y=\sum_{q=0}^{3}Y_q\). The limbs are ordered by decreasing magnitude; terms with \(r+s\geq 4\) are discarded, and periodic renormalization converts the unevaluated sum back into ordered, non-overlapping limbs.

## Key formulas

- $$a_{i,k}=(a_{i,k,0},a_{i,k,1},a_{i,k,2},a_{i,k,3}),\quad b_{k,j}=(b_{k,j,0},b_{k,j,1},b_{k,j,2},b_{k,j,3})$$
- $$A_{i,k}=\left[\begin{array}{c}a_{i,k,0}\\a_{i,k,1}\\a_{i,k,2}\\a_{i,k,3}\end{array}\right]^T,\quad B_{k,j}=\left[\begin{array}{cccc}b_{k,j,0}&b_{k,j,1}&b_{k,j,2}&b_{k,j,3}\\0&b_{k,j,0}&b_{k,j,1}&b_{k,j,2}\\0&0&b_{k,j,0}&b_{k,j,1}\\0&0&0&b_{k,j,0}\end{array}\right]$$
- $$C_{i,j}=\left[\begin{array}{l}a_{i,k,0}b_{k,j,0}\\a_{i,k,1}b_{k,j,0}+a_{i,k,0}b_{k,j,1}\\a_{i,k,2}b_{k,j,0}+a_{i,k,1}b_{k,j,1}+a_{i,k,0}b_{k,j,2}\\a_{i,k,3}b_{k,j,0}+a_{i,k,2}b_{k,j,1}+a_{i,k,1}b_{k,j,2}+a_{i,k,0}b_{k,j,3}\end{array}\right]$$
- $$Y_q=\sum_{r+s=q}W_rX_s,\quad Y^{\mathrm{MD}}=\sum_{q=0}^{3}Y_q$$

## Implementation notes

Integrate the method into one numerically sensitive matrix multiplication, such as a Transformer output projection, attention score product, or the computation of Adam second moments. Store each tensor as four same-shaped limb tensors \((Z_0,Z_1,Z_2,Z_3)\), ordered from largest to smallest magnitude. Initialize with \(Z_0=\operatorname{cast}(Z)\), then form residual limbs by subtracting the represented prefix and periodically renormalizing. For a linear layer, use this pseudocode: `for q in 0..3: Y[q] = 0; for r in 0..q: s = q-r; Y[q] += matmul_tensorcore(W[r], X[s])`; then return `Y_value = Y[0] + Y[1] + Y[2] + Y[3]`. Distinct limb pairs may be grouped into batched tensor-core GEMMs; the triangular Toeplitz arrangement is an alternative packing. Do not renormalize inside the MMA loop. After every fixed block of operations, renormalize each element by sorting limb magnitudes and repeatedly splitting a sum into a rounded component and residual, for example using `s = fl(x+y)`, `e = (x-(s-y)) + (y-(s-x))`. Estimate truncation error on sampled tiles against FP64 GEMMs and measure tensor-core overhead separately. Start with a 6-layer Transformer on a long-context synthetic task and a small language dataset. Compare BF16, BF16 with FP32 accumulation, and four-limb BF16 accumulation at equal batch size. Record throughput, loss spikes, gradient overflow or underflow, and validation loss. The method succeeds if it reduces instability or improves validation loss on long reductions while retaining useful throughput, with periodic renormalization outperforming per-operation renormalization in the speed-stability tradeoff.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.