# Initial-Only Weight Decay with Tail Averaging

- ID: 85
- Canonical URL: https://synthcore.org/idea/85/initial-only-weight-decay-with-tail-averaging
- API JSON: https://synthcore.org/api/idea/85.json
- API Markdown: https://synthcore.org/api/idea/85.md
- Verification status: audited
- Source: [arXiv:2608.22953](https://arxiv.org/abs/2608.22953)
- Category: optimization
- Solves: stability, generalization, accuracy
- ML areas: optimizer, regularization, training-dynamics
- Math tags: optimization, spectral-theory, stochastic-processes, linear-algebra
- Ratings: usefulness 6/10; difficulty 3/10; novelty 4/10

## Idea description

Replace constant weight decay by the paper's initial-regularization schedule: apply isotropic shrinkage only for the first m optimizer steps, then set the regularization coefficient exactly to zero. Collect checkpoints only in a later interval, such as steps 2m through 3m, and average them to reduce stochastic variance. This creates an explicitly separated representation-shaping phase and fitting phase rather than forcing one regularization strength throughout training.

## Mathematical statement

The core SGDIR recursion is equation (1): \(\theta_{t+1}=(1-\gamma\lambda_t)\theta_t-\gamma(\langle x_t,\theta_t\rangle-y_t)x_t\), where \(\theta_t\) is the parameter vector, \(x_t\) is the sampled feature/input vector, \(y_t\) is its target, \(\gamma>0\) is the constant step size, and \(\lambda_t=\Lambda\mathbf{1}(t<m)\) applies regularization of strength \(\Lambda>0\) only before step \(m\). The paper analyzes the tail average \(\bar{\theta}_{2m,3m}=m^{-1}\sum_{t=2m}^{3m-1}\theta_t\). For the error relative to the population minimizer \(\theta^*\), equation (37) gives \(w_{t+1}-\theta^*=P_t(w_t-\theta^*)-\gamma\lambda_t\theta^*\), with \(P_t=I-\gamma(x_t\otimes x_t+\lambda_t I)\). Here \(x_t\otimes x_t\) is the rank-one covariance operator and \(w_t\) denotes the regularized iterate. If \(\Sigma=\mathbb{E}[x_t\otimes x_t]\), then \(A_t=\mathbb{E}[P_t]=I-\gamma(\Sigma+\lambda_t I)\). Lemma 9 states \(A_t\succeq0\) and \(\mathbb{E}[P_t^2]\preceq(1-\gamma\lambda_t)A_t\preceq(1-\gamma\lambda_t)^2I\). The adaptation applies the same multiplicative shrinkage to neural-network weights, treating the local stochastic update as the nonlinear analogue of the linear recursion; the inequality motivates choosing \(\gamma\) so that the initial phase has \(0\leq\gamma\Lambda\leq1\), while ordinary gradient stability is controlled empirically or by a short warmup.

## Key formulas

- $$\theta_{t+1}=(1-\gamma\lambda_t)\theta_t-\gamma\big(\langle x_t,\theta_t\rangle-y_t\big)x_t,\qquad \lambda_t=\Lambda\mathbf{1}(t<m).$$
- $$\bar{\theta}_{2m,3m}=\frac{1}{m}\sum_{t=2m}^{3m-1}\theta_t.$$
- $$w_{t+1}-\theta^{*}=P_t(w_t-\theta^{*})-\gamma\lambda_t\theta^{*},\qquad P_t=I-\gamma(x_t\otimes x_t+\lambda_t I).$$
- $$\mathbb{E}[P_t^2]\preceq(1-\gamma\lambda_t)A_t\preceq(1-\gamma\lambda_t)^2I,\qquad A_t=I-\gamma(\Sigma+\lambda_t I),\quad \Sigma=\mathbb{E}[x_t\otimes x_t].$$

## Implementation notes

1. Integration point: modify the SGD or momentum-SGD parameter update, immediately after computing the minibatch gradient and before the parameter update. Exclude biases, normalization affine parameters, and optionally embeddings from shrinkage, matching common decoupled weight-decay practice. Let \(m\) be the initial-regularization horizon and \(\Lambda\) the decay coefficient. 2. Pseudocode: initialize parameters \(\theta\) with the chosen network initialization; for optimizer step \(t=0,\ldots,T-1\), compute minibatch gradient \(g_t\), set \(\lambda_t=\Lambda\) if \(t<m\), otherwise set \(\lambda_t=0\), and perform \(\theta\leftarrow(1-\gamma\lambda_t)\theta-\gamma g_t\). Store parameter checkpoints for \(t\in[2m,3m)\), and at the end set \(\theta_{out}=m^{-1}\sum_{t=2m}^{3m-1}\theta_t\). In a deep network, maintain the running average in FP32 and evaluate the averaged copy after updating batch-normalization statistics. 3. Mathematical quantities: \(\Lambda,m,\gamma\), the exact shrinkage factor \(1-\gamma\Lambda\), and the tail-average window come directly from the construction. The covariance operator \(\Sigma\), eigenvalues, and theoretical excess-risk constants need not be estimated for the MVP. Estimate stability empirically using minibatch gradient norms and the largest observed parameter-update ratio; require \(0\leq\gamma\Lambda\leq1\) and clip if violated. 4. First experiment: train a 2-layer MLP and a small ResNet-18 on CIFAR-10, plus a linear or kernelized regression model where the assumptions are closest to the paper. Compare constant weight decay, cosine-decayed decay, no decay, and initial-only decay at matched learning rate, total steps, and batch size. Sweep \(m/T\in\{0.05,0.1,0.25,0.5\}\) and \(\gamma\Lambda\in\{0.01,0.05,0.1,0.5\}\). Measure training loss, validation accuracy, sharpness proxy, gradient norm spikes, and final test accuracy. Success means lower validation error or better late-stage loss than constant decay, with fewer early instability spikes and no extra forward/backward FLOPs; additionally test whether tail averaging improves accuracy at fixed compute.

## Verification

- Status: audited
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a CPU-only SGDIR-style linear regression MVP with exact initial-only shrinkage, delayed tail averaging, and a numerical diagonal-operator check. The core inequality passed, with maximum estimated violation -0.000919, and initial-only decay strongly beat constant decay on test MSE (0.06390 vs 0.32036). However, no decay was marginally better (0.06373), so the evidence supports avoiding persistent decay more clearly than proving an advantage over no regularization.

### Mechanism check

- Verdict: Built a CPU-only SGDIR-style linear regression MVP with exact initial-only shrinkage, delayed tail averaging, and a numerical diagonal-operator check. The core inequality passed, with maximum estimated violation -0.000919, and initial-only decay strongly beat constant decay on test MSE (0.06390 vs 0.32036). However, no decay was marginally better (0.06373), so the evidence supports avoiding persistent decay more clearly than proving an advantage over no regularization.
- Confidence: 7/10
- Limitations: Only synthetic linear squared-loss regression was tested; no MLP, CIFAR-10, momentum, hyperparameter tuning, FLOP benchmarking, sharpness measurement, or tuned constant-decay comparison was included. The operator check uses finite Monte Carlo sampling.

## Artifacts

- [exp85_initial_only_weight_decay_tail_averaging.py](https://synthcore.org/code/24/exp85_initial_only_weight_decay_tail_averaging.py)
- [experiment.py](https://synthcore.org/code/24/experiment.py)
- [report.md](https://synthcore.org/code/24/report.md)
- [report_super_2026-08-30T083749.md](https://synthcore.org/code/24/report_super_2026-08-30T083749.md)
- [results.json](https://synthcore.org/code/24/results.json)
- [super_results.json](https://synthcore.org/code/24/super_results.json)
- [Download all files as ZIP](https://synthcore.org/download/24)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.