# High-precision accumulation with low-precision ODE stages

- ID: 509
- Canonical URL: https://synthcore.org/idea/509/high-precision-accumulation-with-low-precision-ode-stages
- API JSON: https://synthcore.org/api/idea/509.json
- API Markdown: https://synthcore.org/api/idea/509.md
- Verification status: unverified
- Source: [arXiv:2607.07080](https://arxiv.org/abs/2607.07080)
- Category: sampling
- Solves: speedup, stability
- ML areas: diffusion-sampling, ssm, vae
- Math tags: numerical-analysis, dynamical-systems
- Ratings: usefulness 5/10; difficulty 4/10; novelty 3/10

## Idea description

Use low precision only for repeated neural-function evaluations and intermediate stage vectors of an explicit ODE solver, while keeping the current state, timestep scaling, and final weighted accumulation in higher precision. This targets neural ODEs and diffusion probability-flow samplers, where function evaluations dominate runtime but accumulated integration error can destabilize long trajectories.

## Mathematical statement

The paper considers the ODE $\dot{y}(t)=f(t,y(t))$ with $y(t_0)=y_0$, and a general $q$-stage explicit method $y_{i+1}=y_i+h\sum_{l=1}^{q}a_l k_{l,i}$, where $h=(T-t_0)/N$, $y_i$ approximates $y(t_i)$, and $k_{l,i}=f(t_{l,i},y_{l,i})$. The transferable construction is a mixed-arithmetic realization of this recurrence: compute selected stage states and evaluations in low precision, but perform the outer state update and summation in high precision. Let $p_H$ and $p_L$ denote high and low arithmetic precision, $\operatorname{cast}_{p}$ a cast to precision $p$, and $\operatorname{fl}_{p}$ a rounded operation in precision $p$. The adapted update is $\widehat{k}_{l,i}=\operatorname{cast}_{p_L}(f_{\theta}(t_{l,i},\operatorname{cast}_{p_L}(y_{l,i})))$ followed by $y_{i+1}=\operatorname{fl}_{p_H}\left(y_i+h\sum_{l=1}^{q}a_l\operatorname{cast}_{p_H}(\widehat{k}_{l,i})\right)$. The stability mechanism is that rounding noise from individual stages is prevented from accumulating through the final state and weighted sum in low precision; the paper also reports that smaller $h$ improves robustness of the mixed method.

## Key formulas

- $$\dot{y}(t)=f(t,y(t)),\qquad t_0\leq t\leq T,\qquad y(t_0)=y_0.$$
- $$y_{i+1}=y_i+h\sum_{l=1}^{q}a_l k_{l,i},\qquad k_{l,i}=f(t_{l,i},y_{l,i}),\qquad h=\frac{T-t_0}{N}.$$
- $$\widehat{k}_{l,i}=\operatorname{cast}_{p_L}\left(f_{\theta}\left(t_{l,i},\operatorname{cast}_{p_L}(y_{l,i})\right)\right),\qquad y_{i+1}=\operatorname{fl}_{p_H}\left(y_i+h\sum_{l=1}^{q}a_l\operatorname{cast}_{p_H}(\widehat{k}_{l,i})\right).$$

## Implementation notes

Integrate this at the ODE solver used by a neural ODE, continuous-depth block, or diffusion probability-flow sampler, specifically around repeated calls to the learned vector field $f_\theta$. Use float32 as $p_H$ and float16 or bfloat16 as $p_L$ initially. Keep the solver's global state $y_i$, timestep $h$, coefficients $a_l$, stage-time arithmetic, and final accumulation in float32; cast only each stage input to low precision and execute the neural network under autocast. For a general explicit method, use this pseudocode: input $y_i,t_i,h$; initialize float32 accumulator $s=0$; for $l=1,\ldots,q$, construct the prescribed stage state $y_{l,i}$ from earlier stages, form $u_l=\operatorname{cast}_{p_L}(y_{l,i})$, evaluate $k_{l,i}=f_\theta(t_{l,i},u_l)$ under low precision, cast $k_{l,i}$ back to float32, and update $s\leftarrow s+a_l k_{l,i}$; return $y_{i+1}=y_i+h s$. If stage states depend on previous stages, construct them in float32 and cast only at the network boundary. The mathematical quantities taken from the paper are the explicit-stage recurrence and the precision partition; no unknown theorem constants are required. Measure function-evaluation throughput, trajectory error against an all-float64 reference, gradient error if training, NaN/Inf frequency, and sensitivity to step size. First test a small neural ODE on MNIST or CIFAR-10 and a VP diffusion probability-flow ODE on CIFAR-10, comparing all-float32, all-float16, and mixed float16-stage/float32-accumulation at identical solver steps. Success means lower wall-clock time or energy at the same endpoint error, substantially fewer failures than all-float16, and no material accuracy loss versus float32.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.