# Эксперимент: Initial-Only Weight Decay with Tail Averaging (#24)

{
  "worked": true,
  "confidence": 7,
  "verdict": "Built a CPU-only SGDIR-style linear regression MVP with exact initial-only shrinkage, delayed tail averaging, and a numerical diagonal-operator check. The core inequality passed, with maximum estimated violation -0.000919, and initial-only decay strongly beat constant decay on test MSE (0.06390 vs 0.32036). However, no decay was marginally better (0.06373), so the evidence supports avoiding persistent decay more clearly than proving an advantage over no regularization.",
  "metrics": {
    "baseline": "Constant decay: final test MSE 0.3204 +/- 0.0714; tail MSE 0.3126 +/- 0.0714 across 8 seeds.",
    "idea": "Initial-only decay: final test MSE 0.06390 +/- 0.00144; tail-averaged MSE 0.06370 +/- 0.00151. No-decay reference: final 0.06373; tail 0.06328."
  },
  "how_to_run": "/home/maxwelhelp/main/bin/python3 experiment.py --runs 8 --out results.json",
  "files": [
    "experiment.py",
    "results.json"
  ],
  "limitations": "Only synthetic linear squared-loss regression was tested; no MLP, CIFAR-10, momentum, hyperparameter tuning, FLOP benchmarking, sharpness measurement, or tuned constant-decay comparison was included. The operator check uses finite Monte Carlo sampling."
}

## Super-check round: specified MNIST AdamW experiment

Implemented and ran `exp85_initial_only_weight_decay_tail_averaging.py` with the supervisor's four modes on real MNIST (60,000 train / 10,000 test), normalized 28x28 inputs, a 784-128-128-10 ReLU MLP, AdamW (`lr=1e-3`, betas `(0.9, 0.999)`), cosine annealing, batch 128, 3,000 optimizer steps, `weight_decay=0.05`, initial horizon `p=0.5`, and tail window `q=0.2`. Five seeds (0--4) ran on CUDA; total runtime was 461.8 seconds (about 23 seconds per mode/seed). The implementation includes baseline, initial-only WD, initial-only plus tail averaging, and constant WD plus tail averaging, as well as test accuracy, best accuracy, steps-to-best, weight norm, mid-training gradient norm, and runtime.

The operator sanity check passed: gamma-lambda product 0.14, maximum Monte Carlo gap for the first inequality -0.000371, scalar-bound gap -0.000376.

Results (mean +/- sample standard deviation across five seeds):

| mode | test accuracy | best accuracy | steps to best | weight norm | mid gradient norm | runtime sec |
|---|---:|---:|---:|---:|---:|---:|
| baseline (constant WD) | 0.97720 +/- 0.00106 | 0.97764 +/- 0.00058 | 2700 +/- 300 | 15.5723 +/- 0.1096 | 0.5346 +/- 0.0108 | 23.06 +/- 1.37 |
| initwd | 0.97724 +/- 0.00103 | 0.97764 +/- 0.00059 | 2700 +/- 300 | 15.7558 +/- 0.1114 | 0.5352 +/- 0.0105 | 21.96 +/- 1.31 |
| initwd_tail (idea) | 0.97744 +/- 0.00110 | 0.97764 +/- 0.00059 | 2700 +/- 300 | 15.7558 +/- 0.1114 | 0.5352 +/- 0.0105 | 23.62 +/- 1.25 |
| wd_tail | 0.97744 +/- 0.00109 | 0.97764 +/- 0.00058 | 2700 +/- 300 | 15.5723 +/- 0.1096 | 0.5346 +/- 0.0108 | 23.65 +/- 1.46 |

Paired test-accuracy differences were initwd_tail-baseline = +0.00024 +/- 0.00025 (4/5 wins), initwd_tail-initwd = +0.00020 +/- 0.00032 (4/5 wins), and initwd_tail-wd_tail = approximately 0.00000 +/- 0.00019 (2/5 wins). Thus the apparent +0.024 percentage-point gain over baseline is exactly reproduced by constant-WD tail averaging; it is not evidence for initial-only decay. Initial-only WD without averaging is essentially tied with baseline. Tail averaging adds roughly 0.00024 accuracy but requires extra parameter accumulation and evaluation, and no faster best step was observed.

**Super-check verdict: worked=false.** The mathematical check remains valid and the implementation is operational, but the promised combined effect does not survive component isolation on this harder setup: `wd_tail` matches `initwd_tail`, while `initwd` is tied with the constant-decay baseline. This is a negative/neutral result rather than a meaningful win for initial-only weight decay.