✗ Failed on benchmark 2026

Centered Heavy-Tail Clipping Optimizer

Implementation & benchmark of arXiv:2609.03990 — Heavy-Tailed First-Order Optimization for Polyak-Łojasiewicz Condition: High-Dimensional Minimax Bounds, High-Probability Guarantee, and Fixed-Dimensional Improvements

Usefulness7/10

Difficulty4/10

Novelty6/10

Optimization Stability Speedup Generalization Optimizer Training dynamics Regularization Optimization Probability Convex analysis Dynamical systems

Source paper: Heavy-Tailed First-Order Optimization for Polyak-Łojasiewicz Condition: High-Dimensional Minimax Bounds, High-Probability Guarantee, and Fixed-Dimensional Improvements arXiv:2609.03990 ⓘ · analyzed Sep 4, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace ordinary global gradient clipping with clipping of each stochastic gradient around a robust minibatch center rather than around zero. This preserves the common directional component of the gradients and suppresses only heavy-tailed residuals, making the update usable when gradient noise has a finite α-moment for 1 < α ≤ 2 but no finite variance.

Formulas

$$\mathbb{E}\!\left[\|G_t-\nabla f(x_t)\|^{\alpha}\mid\mathcal{F}_{t-1}\right]\leq\sigma^{\alpha},\qquad 1<\alpha\leq 2.$$

$$\frac{1}{2}\|\nabla f(x)\|^{2}\geq\mu\bigl(f(x)-f^{\star}\bigr).$$

$$T_{\epsilon}=\Omega_{\alpha}\!\left[\kappa\log\frac{\Delta_{0}}{\epsilon}+\kappa\left(\frac{\sigma^{2}}{\mu\epsilon}\right)^{\frac{\alpha}{2(\alpha-1)}}\right].$$

$$C_{\tau}(g;c)=c+(g-c)\min\left\{1,\frac{\tau}{\|g-c\|}\right\},\qquad x_{t+1}=\arg\min_x\left\{\langle h_t,x\rangle+\frac{1}{\eta_t}D_{\psi}(x,x_t)\right\}.$$

Mathematical statement

The paper assumes conditionally unbiased oracle gradients G_t with E[G_t | F_{t-1}] = ∇f(x_t) and finite conditional moment E[||G_t - ∇f(x_t)||^α | F_{t-1}] ≤ σ^α, where F_{t-1} is the history before step t, α ∈ (1,2], and σ is the heavy-tail scale. The objective obeys the PL inequality (1/2)||∇f(x)||^2 ≥ μ(f(x)-f★), where μ is the PL constant and f★ is the global minimum value; L-smoothness gives condition number κ=L/μ. The paper's target stochastic complexity is T_ε = Ω_α[κ log(Δ_0/ε) + κ(σ^2/(με))^{α/[2(α-1)]}], with Δ_0=f(x_0)-f★. Use the centered clipping operator C_τ(g;c)=c+(g-c)min{1,τ/||g-c||}, where c is a robust estimate of the current mean gradient and τ is a residual radius. For a mirror map ψ, use x_{t+1}=argmin_x{<h_t,x>+(1/η_t)D_ψ(x,x_t)}, where h_t is the averaged centered-clipped gradient and D_ψ(x,y)=ψ(x)-ψ(y)-<∇ψ(y),x-y> is the Bregman divergence. In Euclidean parameter space this reduces to x_{t+1}=x_t-η_t h_t. Clipping bias is controlled by the finite moment because E[||(G-∇f)1_{||G-∇f||>τ}||] ≤ σ^α/τ^{α-1}; increasing τ therefore trades bias against outlier sensitivity.

Implementation notes

Integrate this at the optimizer update, after per-example gradients are computed and before momentum or parameter updates. For each minibatch B_t, let g_i be the flattened gradient of example i. Compute a robust center c_t using the coordinate-wise median for a cheap baseline, or approximate the geometric median with 3-5 Weiszfeld iterations. Form residuals r_i=g_i-c_t and clip them with h_i=c_t+r_i*min(1,tau/(||r_i||_2+1e-12)). Average h_t=mean_i h_i, optionally apply momentum, and update parameters with Euclidean mirror descent theta=theta-eta*h_t. For Adam compatibility, feed h_t rather than the raw batch gradient into the moment accumulators: m=beta1*m+(1-beta1)*h_t; v=beta2*v+(1-beta2)*h_t*h_t; theta=theta-eta*m/(sqrt(v)+eps). Initialize tau=k*sigma_hat*(|B_t|/log(2/delta))^(1/alpha), with k in {1,2,4}; estimate sigma_hat from the median or 0.8-quantile of residual norms using a slow EMA. If alpha is unknown, sweep alpha in {1.2,1.5,1.8,2}. The paper supplies the finite-moment assumption, PL contraction structure, and target complexity; the center, scale estimator, and layerwise radius are empirical approximations. First test a 2-layer MLP or small ResNet on CIFAR-10 with Student-t gradient perturbations of degrees of freedom 1.2-1.8 and occasional 10-100x outliers. Compare SGD, global norm clipping, AdamW, and this optimizer at equal minibatch size and FLOPs. Measure loss descent, update-norm quantiles, divergence frequency, validation accuracy, and robustness as tails worsen. Success means fewer catastrophic steps and lower loss or higher accuracy at equal compute, while matching standard optimizers on Gaussian-noise controls.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a reproducible NumPy MVP implementing coordinate-median centered residual clipping and verified its core behavior. The clipping operator preserved direction exactly and bounded residuals by tau; the measured tail-bias log-log slope was -0.506 versus the predicted 1-alpha=-0.5. In the quadratic toy experiment, centered clipping reached mean final loss 0.00122 under heavy tails versus 0.0563 for global clipping and 1.59 for unclipped SGD, while also improving the Gaussian control.

Agent confidence: 8/10

Baseline: Heavy-tail final loss: SGD 1.5905, global clipping 0.05627; Gaussian global clipping 0.001683. Heavy-tail global clipping had mean p99 update norm 0.150.
Idea: Centered clipping: heavy-tail final loss 0.001221 over 12 seeds, Gaussian final loss 0.000715; mean p99 update norms 1.329 and 1.329 respectively; zero divergences. Tail-bias slope -0.506 versus predicted -0.5; operator max residual 0.73 for tau=0.73 and minimum direction cosine 1.0.

Limitations: This is a synthetic 20-dimensional quadratic oracle experiment, not a 2-layer MLP or CIFAR-10 test. The center is a coordinate-wise median and tau is fixed rather than adaptively estimated; no Adam compatibility, momentum, mirror-map variant, wall-clock or FLOP accounting, or broader tail, tau, and learning-rate sweep was tested. Global clipping uses the same small tau and therefore clips the common gradient direction, so the comparison demonstrates the proposed mechanism but is not a fully tuned optimizer benchmark.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.03990) Sep 4, 2026

Significantly WORSE than the tuned baseline (+32.0%, p=0.0324, wins 1 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 11.4363
Idea mean: 15.0967
Effect (Δ): 3.6603 (+32.0%; negative = idea better)
Wins: 1 / 8 paired seeds
p-value: 0.0324 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 15 configs

Limitations:

Only the registered tabular optimizer track was tested. No vision, sequence, dynamics, adaptive tau, geometric median, SGD comparison, wall-clock/FLOP accounting, or broader tail/control sweep was run. Explicit per-example gradients make the implementation computationally expensive.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 10/10

The centered heavy-tail clipping optimizer was tested on the registered tabular Friedman#1 benchmark with the shared mlp_tiny architecture and 8 paired seeds. The tuned baseline achieved mean test MSE 11.4363, while the best centered-clipping configuration achieved 15.0967; the idea was significantly worse, with paired permutation p=0.03245. The trained-model mechanism signature was not confirmed because the observed mean directional cosine was 0.5657, so there is no benchmark win.

Baseline: Adam with global norm clipping; best lr=0.024, tau=1.0; mean MSE 11.4363187551.
Idea: Adam with coordinate-median centered residual clipping; best lr=0.024, tau=1.0; mean MSE 15.0966548920.
Mechanism signature: Residual bound matched tau numerically, but observed directional cosine mean was 0.5656541213; confirmed=false.

Benchmark result