Training-Free Universal Approximation by Prompting Random Transformers

What the math gives to ML

The paper gives a constructive separation between stored weights and task behavior: a frozen, randomly initialized single-layer softmax transformer can implement a kernel regressor when its prompt is chosen by solving linear systems. The transferable asset is the explicit conversion of attention logits into Gaussian-kernel exponents, turning prompt construction into linear algebra rather than gradient-based fine-tuning. This suggests testing random frozen transformers as cheap task-specific adapters, with prompts containing support points and labels and with prompt length controlling approximation quality. The main practical risk is that the required prompt norm or length may be too large for realistic transformer dimensions.