# Kernel-Prompted Random Transformer

- ID: 2052
- Canonical URL: https://synthcore.org/idea/2052/kernel-prompted-random-transformer
- API JSON: https://synthcore.org/api/idea/2052.json
- API Markdown: https://synthcore.org/api/idea/2052.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.09558](https://arxiv.org/abs/2608.09558)
- Category: architecture
- Solves: memory, sample-efficiency, inference-speedup
- ML areas: transformer, attention, fine-tuning, inference-speedup
- Math tags: approximation-theory, linear-algebra, statistics, probability
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Freeze a randomly initialized single-layer transformer and use a constructed soft prompt to make its attention weights equal Gaussian-kernel weights over support examples. The resulting model performs Nadaraya-Watson regression in one forward pass, so task adaptation stores prompt tokens rather than modifying network weights. Prompt length becomes the number of kernel centers, while hidden dimension and prompt norm determine whether the required logits can be represented accurately.

## Mathematical statement

The paper embeds a query x in R^{d_in} as x_emb = [x^T,1,0,...,0] in R^{1 x d_h}, where d_in is the input dimension and d_h >= d_in+2 is the hidden dimension. For support points x_i in R^{d_in}, labels y_i in R^m, and bandwidth sigma > 0, the target attention logits are b_i(x) = -||x-x_i||_2^2/(2 sigma^2). The desired predictor is the Nadaraya-Watson estimator f_hat(x) = sum_i exp(b_i(x)) y_i / sum_j exp(b_j(x)). Let q(x) be the frozen query vector and k_i be the key vector generated by prompt token p_i; the actual attention logit is ell_i(x) = q(x)^T k_i/sqrt(d_h), up to a fixed temperature. Expanding the squared distance gives b_i(x) = -||x||^2/(2 sigma^2) + x^T x_i/sigma^2 - ||x_i||^2/(2 sigma^2). The query-dependent common term -||x||^2/(2 sigma^2) cancels in softmax, so the prompt only needs to match the affine coefficients of x and the constant term. Prompt construction solves M z_i = c_i, where z_i contains free prompt coordinates, M maps those coordinates to coefficients of ell_i(x), and c_i contains the desired coefficients x_i/sigma^2 and -||x_i||^2/(2 sigma^2). The required rank condition is that M has full row rank; under Gaussian random weights this holds almost surely when d_h is sufficiently large. If exact matching is unstable, use ridge regression.

## Key formulas

- $$E(\mathbf{x})=\mathbf{x}_{\mathrm{emb}}\coloneqq[\mathbf{x}^{T},1,0,\ldots,0]\in\mathbb{R}^{1\times d_{\mathrm{h}}},\qquad d_{\mathrm{h}}\ge d_{\mathrm{in}}+2.$$
- $$b_i(\mathbf{x})=-\frac{\|\mathbf{x}-\mathbf{x}_i\|_2^2}{2\sigma^2},\qquad \widehat f(\mathbf{x})=\frac{\sum_{i=1}^{N}\exp(b_i(\mathbf{x}))\,\mathbf{y}_i}{\sum_{j=1}^{N}\exp(b_j(\mathbf{x}))}.$$
- $$\ell_i(\mathbf{x})=\frac{q(\mathbf{x})^{T}k(\mathbf{p}_i)}{\sqrt{d_h}},\qquad Mz_i=c_i\ \text{with}\ c_i\ \text{chosen so that}\ \ell_i(\mathbf{x})=b_i(\mathbf{x})\ \text{up to an additive term common to all }i.$$
- $$z_i^{\star}=\arg\min_{z}\|Mz-c_i\|_2^2+\lambda\|z\|_2^2=(M^{T}M+\lambda I)^{-1}M^{T}c_i.$$

## Implementation notes

1. Integration point: implement a frozen one-layer attention module with randomly sampled Gaussian query, key, value, and output matrices. Reserve N prompt slots followed by one query slot. Encode each support example (x_i,y_i) into a prompt token, exposing enough hidden coordinates to control its key and value. Encode the query as [x^T,1,0,...,0]. Read the output at the query position and decode it as the prediction.

2. Pseudocode:
```text
sample frozen Q,K,V,O ~ Normal(0, 1/d_h)
choose support set {(x_i,y_i)} and bandwidth sigma
for each support i:
    construct target coefficients c_i from
        b_i(x)=-(||x||^2 - 2*x^T*x_i + ||x_i||^2)/(2*sigma^2)
    probe the frozen logit ell_i(x; z) on d_in+2 basis query embeddings
    construct M mapping free prompt coordinates z to logit coefficients
    z_i = solve_ridge(M, c_i, lambda)
    write z_i into prompt token p_i
    choose remaining value coordinates so V(p_i) decodes to y_i
for query x:
    run frozen attention on [p_1,...,p_N,E(x)]
    return decoded query-position output
```

3. Compute M analytically from the frozen linear query/key maps when possible; otherwise estimate it with finite differences around a zero prompt. Use SVD to measure rank and condition number. If the smallest singular value is small, use the ridge solution above and record the residual ||Mz_i-c_i||_2. Estimate sigma using validation data or sigma equal to a multiple of the median pairwise support distance. The paper provides the full-rank/almost-sure-randomness rationale, but the implementation must empirically measure prompt norm, logit residual, attention-weight error relative to the Gaussian kernel, and final prediction error. If key and value coordinates are coupled, solve one joint least-squares system that matches both logit coefficients and the decoded labels.

4. First experiment: compare analytical prompting against gradient-optimized soft prompts, a frozen random transformer without prompting, k-nearest neighbors, and direct Nadaraya-Watson regression. Start with a two-dimensional synthetic Holder function and 32, 64, 128, and 256 support tokens; then test MNIST or CIFAR-10 embeddings. Sweep hidden dimension, bandwidth, ridge coefficient, and prompt length. Measure adaptation time, inference latency, prompt memory, test MSE or accuracy, logit residual, and prompt norm. Success means reaching kernel-regression accuracy with no backpropagation during adaptation, lower adaptation cost than prompt tuning, and decreasing error as prompt length and hidden dimension increase.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built an analytical frozen random-attention prompt constructor that jointly matches Gaussian-kernel logit coefficients and support labels. The mechanism manifested: rank became full at dh=4=d_in+2 with residual below 1.4e-14; kernel attention weight error was below 4.4e-15; prompt norms scaled approximately as sigma^-2, with observed ratios 1.96, 2.04, and 2.01 versus predicted 1.96, 2.04, and 2.04. Ridge regularization reduced prompt norm from 45.56 to 1.26 while residual increased from approximately 0 to 7.28. The method exactly reproduced Nadaraya-Watson regression, but did not beat 3-NN on this toy task, so this validates the mechanism rather than a general predictive win.

### Mechanism check

- Verdict: Built an analytical frozen random-attention prompt constructor that jointly matches Gaussian-kernel logit coefficients and support labels. The mechanism manifested: rank became full at dh=4=d_in+2 with residual below 1.4e-14; kernel attention weight error was below 4.4e-15; prompt norms scaled approximately as sigma^-2, with observed ratios 1.96, 2.04, and 2.01 versus predicted 1.96, 2.04, and 2.04. Ridge regularization reduced prompt norm from 45.56 to 1.26 while residual increased from approximately 0 to 7.28. The method exactly reproduced Nadaraya-Watson regression, but did not beat 3-NN on this toy task, so this validates the mechanism rather than a general predictive win.
- Confidence: 9/10
- Limitations: No MNIST/CIFAR embedding test, multi-head or full transformer block, GPU benchmark, gradient-optimized soft-prompt comparison, or random-unprompted transformer baseline was tested. The implementation models the frozen attention algebra with Gaussian linear Q/K/V maps rather than a full tokenized transformer forward pass.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/826/bench_report.json)
- [kernel_prompt_experiment.py](https://synthcore.org/code/826/kernel_prompt_experiment.py)
- [report.md](https://synthcore.org/code/826/report.md)
- [report_bench_2026-09-01T112618.md](https://synthcore.org/code/826/report_bench_2026-09-01T112618.md)
- [results.json](https://synthcore.org/code/826/results.json)
- [stage2_bench.py](https://synthcore.org/code/826/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/826)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.