Эксперимент: Exact energy-preserving activation subsampling (#1190)

{ "worked": true, "confidence": 9, "verdict": "Built an exact Hadamard-sign energy estimator for a 16-dimensional modeled coefficient vector, plus a matched random-sign baseline and toy normalized MLP regression experiment. The matrix residual was exactly 0 and energy error was 6.1e-16, while random sampling had 0.323 relative RMSE and nonzero gradient variance; at 220 steps, exact normalization reached 0.00140 loss versus 0.0326 for random sampling at essentially equal runtime. This is a clear signal for the represented subspace, though not evidence of broad neural-network gains.", "metrics": { "baseline": "Random 16-sign estimator: relative energy RMSE 0.3232, mean relative estimator std 0.3223, mean gradient element variance 4.06e-7, final 20-step MLP loss 0.03258, runtime 1.0097 s", "idea": "16-column Hadamard exact estimator: matrix Frobenius residual 0, maximum relative energy error 6.06e-16, mean gradient element variance 3.68e-19, final 20-step MLP loss 0.001396, runtime 1.0264 s; float16 input with float32 accumulation max relative error 7.87e-4" }, "how_to_run": "/home/maxwelhelp/main/bin/python3 experiment.py", "files": [ "experiment.py", "results.json" ], "limitations": "Only a tiny synthetic regression task and N=16 full-support Hadamard design were tested; the proposed 16-of-256 setting, learned projection/complement, CIFAR/WikiText/transformer or MoE workloads, wall-clock memory savings, importance sampling, and larger-scale generalization were not tested. Randomness is in the sign design rather than data minibatching." }