Benign-Misfit Large-Step Phase

Integrate this as an optimizer schedule for a small Transformer, MLP, or linear-probe fine-tuning run, rather than changing the forward architecture. Use two phases. In phase 1, initialize from the pretrained model or zero-initialize a new linear head and perform one shuffled pass over the data with a constant learning rate \(\eta_0\) substantially larger than the usual fine-tuning rate. In phase 2, switch to ordinary AdamW or SGD after the first validation minimum, or after a fixed fraction of an epoch. A practical controller is: (1) estimate a characteristic curvature \(\widehat{\gamma}\) from the top eigenvalue of the head-feature covariance using 5 power iterations; (2) set \(\eta_0=c/\widehat{\gamma}\), with \(c\in\{0.25,0.5,0.9,1.2\}\), so the idealized residual multiplier is \(1-c\); (3) process each minibatch once without recycling it; (4) record train and validation loss after every 1/20 epoch; (5) stop the large-step phase at the first validation minimum or when validation loss rises for three measurements while train loss continues falling. Pseudocode: gamma_hat=power_iteration(X_head.T @ X_head / N); eta=c/gamma_hat; for batch in shuffled_loader: loss=CE(model(batch)); update_SGD(model, loss, eta); evaluate(); if val_loss_worsens: break; continue_with_AdamW(). The paper's exact mathematics is the scalar decay \(b_{t+1}=(1-\eta\gamma)b_t\); in a neural network, \(\widehat{\gamma}\) and the stopping point are empirical proxies, and the model has no literal orthogonal nuisance decomposition. First test on CIFAR-100 or a 10k-example language-model fine-tuning subset using a 2-layer MLP or 2-layer Transformer, comparing standard cosine-decayed AdamW, one-pass ordinary SGD, and this schedule at equal optimizer-update count. Success means lower validation loss or higher accuracy at the point where train loss is still measurably worse than the interpolating baseline, with no increase in instability; a particularly strong signal is faster early validation improvement at equal FLOPs and improved performance in the low-data regime.