Quantile-Winsorized Gradient Updates

Integrate the method at the minibatch gradient aggregation point, before momentum, Adam moments, weight decay, or the parameter update. Start with a small MLP or ResNet on MNIST or CIFAR-10, where per-example gradients can be computed with vectorized Jacobians or microbatching. For a batch of size B, compute g_i=gradient_theta ell_i for every example and flatten each gradient into coordinates j=1,...,P. Choose a winsorization level tau, initially 0.1, and compute q_j^-=Quantile_tau(g_1j,...,g_Bj) and q_j^+=Quantile_(1-tau)(g_1j,...,g_Bj). Replace each g_ij by min(max(g_ij,q_j^-),q_j^+), average over examples, reshape the result, and pass it to the baseline optimizer. Pseudocode: G = per_example_gradients(losses, theta); lo = quantile(G,tau,dim=batch); hi = quantile(G,1-tau,dim=batch); G = clamp(G,lo,hi); grad = mean(G,dim=batch); optimizer.step(grad). Estimate the contamination level from the fraction of unusually large per-example gradient norms, but keep tau fixed in the first experiment. The paper determines the eta and m dependence; m and moment bounds should be estimated from clean gradient samples. Compare SGD and Adam with their winsorized versions under clean training, random label poisoning, and targeted high-loss examples inserted at 1%, 5%, and 10%. Measure clean accuracy, poisoned-example influence, gradient kurtosis, update spikes, and loss descent at equal FLOPs. Success means smaller update spikes and higher clean accuracy under attack, with less than 5% clean-data degradation.