Эксперимент: Finite-Width NNGP Covariance Stabilizer (#1315)

{ "worked": true, "confidence": 8, "verdict": "Built a training-time NNGP covariance stabilizer for a small ReLU MLP, using the analytic ReLU Gaussian covariance recursion and a detached Frobenius covariance-matching penalty. The numerical check matched Monte Carlo ReLU covariance (0.6775 vs 0.6785) and empirical covariance error scaled as width^-0.542, consistent with the claimed O(n^-1/2) behavior. Across three fixed seeds, regularization reduced held-out calibration covariance deviation from mean 0.0699 to 0.0550 (21.3% lower) and validation MSE from 0.3581 to 0.3481 (2.8% lower), so the effect is promising but modest.", "metrics": { "baseline": "Mean over 3 seeds: validation MSE 0.35808; final covariance deviation 0.06993.", "idea": "Mean over 3 seeds: validation MSE 0.34808; final covariance deviation 0.054997. Covariance deviation improved 21.3% and validation MSE improved 2.8%. Width-scaling log-log slope was -0.542; analytic versus Monte Carlo ReLU covariance was 0.67755 versus 0.67852." }, "how_to_run": "/home/maxwelhelp/main/bin/python3 experiment.py", "files": [ "experiment.py", "results.json" ], "limitations": "Only a 3-layer width-64 MLP and a synthetic regression task were tested; no Transformer, RNN, attention, token-level covariance, GELU, width-128-versus-width-256 comparison, lambda sweep, FLOP-matched comparison, or gradient-variance analysis was performed. The covariance target is analytic and detached, and the experiment does not test the full shared-weight recurrent/attention setting." }