A Sharp Joint Bias-Energy Envelope for Radial Clipping
arXiv:2608.18999
2026
Optimization
2 ideas extracted · analyzed Sep 1, 2026
What the math gives to ML
The paper derives an exact, dimension-free envelope for the two competing effects of radial clipping: removed-gradient bias and retained-gradient energy. The transferable asset is the sharp piecewise constant K_p(alpha,beta), together with its phase transition at alpha = p beta, rather than separate loose bounds for bias and variance. This can turn gradient clipping from a fixed heuristic into a moment-calibrated controller that explicitly prices bias against update energy under heavy-tailed p-moment assumptions. The most promising implementation is an optimizer wrapper that estimates a running p-moment and chooses the clipping radius from a target joint-cost budget.
Ideas from this paper
△ Mechanism confirmed, baseline not beaten
2026
Replace a fixed or percentile-based gradient-clipping threshold with a threshold computed from the exact joint bias-energy envelope. The controller allows the user to specify how expensive removed-gradient bias is relative to retained update energy, while a running p-moment estimate determines the radius needed to satisfy a target joint-cost budget.
Useful8/10
Difficulty4/10
Novelty6/10
✓✓ Beats tuned baseline
2026
Use the envelope's phase transition to choose whether clipping should primarily control update energy or preserve the raw gradient and reduce clipping bias. In the energy-dominated regime, regulate the retained update energy; in the bias-dominated regime, regulate the removed-gradient residual and monitor rare outliers explicitly.
Useful7/10
Difficulty5/10
Novelty5/10