Эксперимент: Learning-Rate-Scaled Weight Decay (#525)

{ "worked": true, "confidence": 9, "verdict": "Built a decoupled AdamW variant whose decay coefficient is scaled by clip(eta/eta_max), plus exact toy verification and a matched three-seed regression experiment. The mechanism manifested quantitatively: log-shrinkage ratios matched predicted learning-rate fractions within 0.08% relative error, schedule cumulative shrinkage matched the recursion to 1.2e-15, lambda scaling was approximately linear, and lambda=0 produced identical updates. In the small regression test, scaled decay improved mean final loss from 0.0585 to 0.0290, though this is not evidence of broad generalization.", "metrics": { "baseline": "Three-seed mean final loss 0.05848 (std 0.02435), loss at step 300 0.07288, final norm 3.4731.", "idea": "Three-seed mean final loss 0.02899 (std 0.00910), loss at step 300 0.04410, final norm 3.5798. Exact-decay sweep observed/predicted log-ratio errors were 0–0.075%; schedule log-shrinkage was -0.6727382872123935 observed versus -0.6727382872123923 predicted; lambda-sweep normalized slopes ranged -3.3571 to -3.3710 versus predicted -3.3564." }, "how_to_run": "/home/maxwelhelp/main/bin/python3 experiment.py", "files": [ "experiment.py", "results.json" ], "limitations": "Only a tiny synthetic linear-regression task was tested, not a Transformer, language modeling, Muon, SGD, multiple schedules, or wall-clock/FLOP efficiency. The optimizer implementation is a minimal AdamW reproduction and does not include parameter-group exclusions or production checkpoint/state handling." }