Эксперимент: Kernel-Prompted Random Transformer (#826)

{ "worked": true, "confidence": 9, "verdict": "Built an analytical frozen random-attention prompt constructor that jointly matches Gaussian-kernel logit coefficients and support labels. The mechanism manifested: rank became full at dh=4=d_in+2 with residual below 1.4e-14; kernel attention weight error was below 4.4e-15; prompt norms scaled approximately as sigma^-2, with observed ratios 1.96, 2.04, and 2.01 versus predicted 1.96, 2.04, and 2.04. Ridge regularization reduced prompt norm from 45.56 to 1.26 while residual increased from approximately 0 to 7.28. The method exactly reproduced Nadaraya-Watson regression, but did not beat 3-NN on this toy task, so this validates the mechanism rather than a general predictive win.", "metrics": { "baseline": "Nadaraya-Watson MSE: 0.1445, 0.1380, 0.1169, 0.1202 for N=32,64,128,256; 3-NN MSE: 0.1025, 0.0346, 0.0167, 0.0114.", "idea": "Analytical prompt matched Nadaraya-Watson exactly: MSE identical at every support size; maximum attention-weight error 1.1e-15; logit RMS residual 2.6e-15 to 2.8e-15; adaptation time 0.59-0.74 ms; prompt storage 512-4096 floats." }, "how_to_run": "/home/maxwelhelp/main/bin/python3 kernel_prompt_experiment.py", "files": [ "kernel_prompt_experiment.py", "results.json", "run_output.txt" ], "limitations": "No MNIST/CIFAR embedding test, multi-head or full transformer block, GPU benchmark, gradient-optimized soft-prompt comparison, or random-unprompted transformer baseline was tested. The implementation models the frozen attention algebra with Gaussian linear Q/K/V maps rather than a full tokenized transformer forward pass." }