Эксперимент: Completely Monotone Multiscale Attention Decay (#1246)

{ "worked": true, "confidence": 7, "verdict": "Built a reproducible finite positive exponential-mixture decay module and toy length-extrapolation experiment. The numerical check showed K(d)>0, strictly decreasing finite differences, and positive alternating-sign derivative expressions through order 4. On a synthetic two-scale target trained at length 32 and evaluated at length 64, the mixture achieved KL 6.6e-7 versus 0.01245 for a learned finite lag table with boundary extrapolation, indicating the proposed smooth multiscale parameterization extrapolates as intended. This is a promising toy signal, not evidence of improved language-model accuracy.", "metrics": { "baseline": "Finite learned lag table: test KL 0.0124517934; train cross-entropy 2.4541333", "idea": "4-component positive exponential mixture: test KL 0.0000006586; train cross-entropy 3.3014403; K(d) minimum 0.0904 and maximum finite-difference dK -0.0002452" }, "how_to_run": "/home/maxwelhelp/main/bin/python3 experiment.py", "files": [ "experiment.py", "results.json" ], "limitations": "Only a synthetic attention-distribution fitting task was tested, not a Transformer, WikiText, copy/retrieval task, ALiBi, RoPE, FLOPs, runtime, entropy, gradient stability, or learned content-dependent attention. The baseline extrapolation used repeated oldest-lag bias, and the mixture had a higher training cross-entropy despite much better length extrapolation; broader seeds and optimization comparisons were not performed." }