Эксперимент: Exponentially Growing Learning Rate with Update-Norm Restarts (#1316)

{ "worked": true, "confidence": 8, "verdict": "Built an update-norm restart SGD implementation and verified the scalar exponential-step product formula to numerical precision, with maximum absolute error 5.6e-17. On an ill-conditioned quadratic, exponential growth without restarts diverged with final loss 1.0e191, while the restart rule triggered once and converged to 7.3e-12. The digits MLP did not show a practical training win: fixed SGD reached 89.8% accuracy versus 43.3% for the tested restart configuration.", "metrics": { "baseline": "Ill-conditioned quadratic without restarts: final loss 1.0035e191; digits MLP fixed SGD: loss 0.9185, accuracy 89.78%.", "idea": "Ill-conditioned quadratic with update-norm restart: final loss 7.334e-12 with 1 restart; digits MLP: loss 2.1459, accuracy 43.33%, 8 restarts, peak learning rate 0.01468; scalar recurrence error 5.55e-17." }, "how_to_run": "/home/maxwelhelp/main/bin/python3 experiment.py", "files": [ "experiment.py", "results.json" ], "limitations": "Only one small digits MLP and one restart hyperparameter configuration were tested; no CIFAR/ResNet, momentum ablation, learning-rate sweep, FLOP-normalized comparison, or broad condition-number study was performed. The neural-network comparison used a tuned fixed learning rate of 0.08 but only one initial rate and tolerance for the restart method." }