# Quantile-Winsorized Gradient Updates

- ID: 1605
- Canonical URL: https://synthcore.org/idea/1605/quantile-winsorized-gradient-updates
- API JSON: https://synthcore.org/api/idea/1605.json
- API Markdown: https://synthcore.org/api/idea/1605.md
- Verification status: unverified
- Source: [arXiv:2607.29532](https://arxiv.org/abs/2607.29532)
- Category: optimization
- Solves: stability, generalization, accuracy
- ML areas: optimizer, training, regularization
- Math tags: statistics, probability, optimization
- Ratings: usefulness 5/10; difficulty 6/10; novelty 5/10

## Idea description

Replace the ordinary minibatch mean gradient by a coordinatewise quantile-winsorized mean. Each parameter-gradient coordinate is clipped to empirical lower and upper quantiles before aggregation, limiting the influence of adversarial examples while retaining all samples and avoiding the discontinuity of hard trimming.

## Mathematical statement

The paper studies the linear model y_i=x_i' beta+u_i, where y_i is the response, x_i is a regressor in R^K, beta is the parameter, and u_i is noise. An adversary observes the clean sample V=(V_1,...,V_n), changes an eta_n fraction of observations, and returns the contaminated sample. Its central rate is eta_n^(1-1/m)+n^(-1/2) under finite m-th moments, where m>1 is the available moment order, eta_n is the contamination fraction, and n is the sample size. We transfer this averaging mechanism to per-example gradients g_i=gradient_theta ell(f_theta(x_i),y_i). For coordinate j, let q_j^- and q_j^+ be empirical quantiles of {g_ij}_{i=1}^B, with trimming level tau greater than eta, and define clip(g_ij,q_j^-,q_j^+)=min(max(g_ij,q_j^-),q_j^+). The update uses the winsorized mean gbar_j^W=B^(-1) sum_i clip(g_ij,q_j^-,q_j^+). The proposed neural-network adaptation is expected to have error scaling O(eta^(1-1/m)+B^(-1/2)) when gradient coordinates have bounded m-th moments; this scaling is a transfer heuristic, not a theorem established for neural-network training by the paper.

## Key formulas

- $$y_i=x_i^{\prime}\beta+u_i,\qquad i=1,\ldots,n.$$
- $$\eta_n^{1-\frac{1}{m}}+n^{-1/2}.$$
- $$g_{ij}^{W}=\operatorname{clip}(g_{ij},q^-_j,q^+_j)=\min\{\max\{g_{ij},q^-_j\},q^+_j\},\qquad \bar g_j^{W}=\frac{1}{B}\sum_{i=1}^{B}g_{ij}^{W}.$$
- $$\theta_{t+1}=\theta_t-\alpha_t\bar g^{W}_t,\qquad \|\bar g^{W}_t-\mathbb{E}[g_t]\|\ \text{expected to scale as}\ O\!\left(\eta^{1-1/m}+B^{-1/2}\right).$$

## Implementation notes

Integrate the method at the minibatch gradient aggregation point, before momentum, Adam moments, weight decay, or the parameter update. Start with a small MLP or ResNet on MNIST or CIFAR-10, where per-example gradients can be computed with vectorized Jacobians or microbatching. For a batch of size B, compute g_i=gradient_theta ell_i for every example and flatten each gradient into coordinates j=1,...,P. Choose a winsorization level tau, initially 0.1, and compute q_j^-=Quantile_tau(g_1j,...,g_Bj) and q_j^+=Quantile_(1-tau)(g_1j,...,g_Bj). Replace each g_ij by min(max(g_ij,q_j^-),q_j^+), average over examples, reshape the result, and pass it to the baseline optimizer. Pseudocode: `G = per_example_gradients(losses, theta); lo = quantile(G,tau,dim=batch); hi = quantile(G,1-tau,dim=batch); G = clamp(G,lo,hi); grad = mean(G,dim=batch); optimizer.step(grad)`. Estimate the contamination level from the fraction of unusually large per-example gradient norms, but keep tau fixed in the first experiment. The paper determines the eta and m dependence; m and moment bounds should be estimated from clean gradient samples. Compare SGD and Adam with their winsorized versions under clean training, random label poisoning, and targeted high-loss examples inserted at 1%, 5%, and 10%. Measure clean accuracy, poisoned-example influence, gradient kurtosis, update spikes, and loss descent at equal FLOPs. Success means smaller update spikes and higher clean accuracy under attack, with less than 5% clean-data degradation.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.