Strongly-convex superwind attention

(1) Integration point: modify the attention-logit computation in a standard Transformer head. For each query token $i$ and key token $j$, set $y_{ij}=q_i-k_j$. Give each head a positive diagonal matrix $H=\operatorname{diag}(\operatorname{softplus}(h)+10^{-4})$ and a learned drift vector $b$; optionally make $b$ a small linear projection of the query. Compute the asymmetric cost $F_\theta(y_{ij})$ and use $-F_\theta(y_{ij})^2/\tau$ as the logit. Implement $\phi_\theta(s,q)$ as a two-layer scalar MLP, initialized near $\phi(s,q)=1+0.1s$, with softplus output so the cost remains positive. (2) Pseudocode: H=softplus(h)+eps; y=Q[:, :, None, :]-K[:, None, :, :]; alpha=sqrt(sum(H*y*y,-1)+eps); beta=sum(b*y,-1); s=beta/alpha; q=sum(b*b/H); phi=phi_mlp(s,q); logits=-(alpha*phi)**2/tau + mask; attn=softmax(logits);. During training sample 32--128 $(x,y)$ pairs per batch, use autodiff to compute phi_s and phi_ss, form g=phi-s*phi_s+(q-s*s)*phi_ss, and add lambda_conv * mean(softplus(eps-g)**2) to the task loss. (3) Computed from the paper's mathematics: the $(\alpha,\beta)$ decomposition and the convexity expression $g$. Estimated empirically: the metric parameters $H,b,\phi$, the barrier weight, and whether $F^2/2$ has a positive Hessian. Every few hundred steps, finite-difference or autodiff-test the smallest eigenvalue of the Hessian of $0.5F(y)^2$ at random directions; reject or clip updates producing eigenvalues below $-10^{-4}$. (4) First experiment: train a 4-layer, 8-head Transformer on WikiText-2, comparing standard dot-product attention, unconstrained asymmetric attention, and this convexity-regularized version at equal parameter count. Measure validation perplexity, attention entropy, fraction of negative-curvature samples, gradient norm spikes, and loss variance. The expected signal is improved validation performance or head specialization over symmetric attention, while the convexity barrier prevents the instability seen in the unconstrained asymmetric baseline.