# KPZ Directed-Polymer Attention

- ID: 1893
- Canonical URL: https://synthcore.org/idea/1893/kpz-directed-polymer-attention
- API JSON: https://synthcore.org/api/idea/1893.json
- API Markdown: https://synthcore.org/api/idea/1893.md
- Verification status: unverified
- Source: [arXiv:2608.06459](https://arxiv.org/abs/2608.06459)
- Category: architecture
- Solves: generalization, stability, sample-efficiency
- ML areas: attention, ssm, moe-routing, regularization
- Math tags: stochastic-processes, probability, dynamical-systems
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace independent Gaussian attention noise or unconstrained token routing with a directed-polymer path distribution over positions and layers. The router aggregates exponentially many monotone paths through temporally correlated random edge scores, producing heavy-tailed but spatially coherent routing and preventing attention from collapsing onto a single token. The paper's t^{2/3} wandering and t^{1/3} free-energy fluctuations become measurable diagnostics and tunable targets rather than hard constraints.

## Mathematical statement

The paper studies the single-particle Green's function G(x,t)=\langle\sigma^{-}_{x}(0)\sigma^{+}_{0}(t)\rangle and defines the normalized spatial distribution p(x,t)=|G(x,t)|^2/\sum_y|G(y,t)|^2. Its center is \langle x(t)\rangle=\sum_x x p(x,t), which wanders on scale O(t^{2/3}); the sample-to-sample fluctuations of the free energy F(t)=-\log\sum_x|G(x,t)|^2 scale as t^{1/3}. At weak bath noise, with noise rate \gamma\ll1, the crossover to this strong-disorder behavior occurs at t_    imes=O(\gamma^{-3/2}). The proposed neural adaptation uses a directed-polymer partition function: for token positions x\in\{1,\ldots,L\} and routing depth t, Z_t(x) is the total weight of all monotone paths ending at x, each path receiving the exponential of its accumulated edge score. The normalized router is p_t(x)=Z_t(x)/\sum_y Z_t(y), and its free energy is F_t=-\log\sum_y Z_t(y). Here s_{t,x}^{\rightarrow} and s_{t,x}^{\leftarrow} are learned or fixed edge logits, \beta is inverse temperature, and \xi_{t,x} is zero-mean random medium noise. The KPZ-inspired diagnostics are Var[\sum_x x p_t(x)]^{1/2}\propto t^{2/3} and Var[F_t]^{1/2}\propto t^{1/3}, compared against the diffusive baselines t^{1/2} and t^{1/2}.

## Key formulas

- $$G(x,t)=\langle\sigma^{-}_{x}(0)\sigma^{+}_{0}(t)\rangle,\qquad p(x,t)=\frac{|G(x,t)|^{2}}{\sum_y|G(y,t)|^{2}},\qquad \langle x(t)\rangle=\sum_x x\,p(x,t).$$
- $$\operatorname{wandering}\big(\langle x(t)\rangle\big)=\mathcal{O}(t^{2/3}),\qquad \operatorname{std}\!\left[-\log\sum_x|G(x,t)|^{2}\right]=\mathcal{O}(t^{1/3}),\qquad t_{\times}=\mathcal{O}(\gamma^{-3/2}).$$
- $$Z_{0}(x)=\exp\!\big(\beta h_{0,x}\big),\qquad Z_{t+1}(x)=\exp\!\big(\beta s_{t,x}^{\rightarrow}\big)Z_t(x-1)+\exp\!\big(\beta s_{t,x}^{\leftarrow}\big)Z_t(x+1),$$
- $$p_t(x)=\frac{Z_t(x)}{\sum_y Z_t(y)},\qquad F_t=-\log\sum_y Z_t(y),\qquad \mu_t=\sum_x x\,p_t(x).$$

## Implementation notes

(1) Integration point: add a residual directed-polymer routing block before a temporal convolution, SSM update, or cross-token value aggregation. For a sequence of length L, maintain positive path masses Z over token positions; the router output is y_t=\sum_x p_t(x)V_x, where V_x are token values or layer states. Use a small number T=4--16 of routing steps rather than the full transformer depth. Edge logits can be produced by a two-layer MLP from adjacent token features: s^{right}_{t,x}=MLP([h_x,h_{x+1}]) and s^{left}_{t,x}=MLP([h_x,h_{x-1}]).

(2) Pseudocode: initialize logZ[0,x]=beta*h0[x]. For t in 0..T-1, sample xi_right[t,x], xi_left[t,x] from a standard normal distribution, set a=beta*(s_right[t,x]+sigma*xi_right[t,x]) and b=beta*(s_left[t,x]+sigma*xi_left[t,x]), then compute logZ_next[x]=logaddexp(logZ[x-1]+a, logZ[x+1]+b), with -infinity at sequence boundaries. Normalize p=softmax(logZ_next), compute y=sum_x p[x]*V[x], and optionally add y to the residual stream. The paper-inspired free energy is F=-logsumexp(logZ_next), and the center is mu=sum_x x*p[x].

(3) The exact mathematical inputs are the normalized path distribution p_t, center mu_t, and free energy F_t. Estimate the effective exponents empirically by running 32--128 random seeds, fitting slopes of log std(mu_t) and log std(F_t) versus log t. Do not assume the exponents are exactly 2/3 and 1/3; use them as diagnostics. Set the noise schedule sigma(t)=sigma_0 for t<=t_cross and sigma_0*(t/t_cross)^(-1/3) afterward, where t_cross=c*gamma^(-3/2), gamma is a user-defined routing-noise parameter and c is tuned on validation data. Add a small entropy floor only if p becomes numerically singular.

(4) First experiment: train a 2-layer character-level language model or Tiny Shakespeare transformer with L=128, comparing standard local attention, Gaussian noisy attention, and this router at matched parameter count and FLOPs. Measure validation perplexity, gradient-norm variance, attention entropy, and sensitivity to token dropout. A successful result is lower perplexity or better corruption robustness at equal compute, together with center wandering closer to a t^{2/3} slope, no rapid router collapse, and stable free-energy variance. An ablation over sigma, beta, and T must determine whether gains come from structured paths rather than merely adding noise.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.