Stable-charge attention kernel
Implementation & benchmark of arXiv:2608.18335 — The Decoherence Exponent: Stable Phase Noise and Constraints on Objective State Reduction
Source paper: The Decoherence Exponent: Stable Phase Noise and Constraints on Objective State Reduction arXiv:2608.18335 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Attach each token or graph node a learned scalar charge q_i and add a fractional stable kernel K_ij = exp(-tau D |q_i-q_j|^alpha) to the interaction mechanism. Constrain 0 < alpha <= 2, the exact range in which the kernel is positive semidefinite for arbitrary finite real charge sets, and optionally make tau layer-dependent to obtain multiscale interactions. This provides a principled alternative to unconstrained learned distance biases and can be used either as an attention-logit bias or as a separate normalized kernel-mixing branch.
Formulas
Mathematical statement
The paper's scale-covariant symmetric Levy lift satisfies E[exp(i xi (L_u-L_0))] = exp(-D u |xi|^alpha), where D >= 0 is the noise scale, u >= 0 is operational time, xi is a real frequency, and 0 < alpha <= 2. Averaging the random unitary exp(i L_u Q) produces decay between charge sectors q_a and q_b at rate Gamma_ab = D |q_a-q_b|^alpha. The induced pairwise kernel is K_u(q_a,q_b) = exp[-D u |q_a-q_b|^alpha]. Schoenberg's theorem gives that K_ij = exp[-tau D |q_i-q_j|^alpha] is positive semidefinite for every finite set of real charges q_1,...,q_n exactly when 0 < alpha <= 2; for alpha > 2, three equally spaced charges provide an analytic obstruction. The transferred asset is therefore a globally valid positive-definite affinity with a learnable fractional exponent. The exponential also gives the entrywise semigroup identity K_ij(tau_1+tau_2) = K_ij(tau_1) K_ij(tau_2).
Implementation notes
1. Integration point: add a stable-charge interaction branch to a Transformer self-attention block before value aggregation. For hidden states H of shape batch x sequence x d, compute one scalar charge per token, q_i = w_q^T h_i + b_q. Store unconstrained parameters rho and a, and define D = softplus(rho) + epsilon and alpha = epsilon + (2 - 2 epsilon) sigmoid(a), with epsilon = 1e-3. Let tau = softplus(tau_raw) be a learned nonnegative layer scale. 2. Pseudocode: q = charge_linear(H).squeeze(-1); dist = abs(q[:, :, None] - q[:, None, :]); stable_bias = -tau * D * dist.pow(alpha); logits = Q @ K.transpose(-1,-2) / sqrt(d_head) + stable_bias; A = masked_softmax(logits); Y = A @ V. Also test a separate normalized kernel branch: W = exp(stable_bias) and Y_stable = (W @ V) / (W.sum(-1, keepdim=True) + eps), followed by a learned gate combining Y and Y_stable. Compute distances and powers in fp32 before casting to the attention dtype. 3. The paper supplies the exponent range, positive-semidefinite kernel guarantee, and entrywise semigroup law. Backpropagation estimates q_i, D, alpha, and tau; no eigenvalue computation is needed. Do not claim that content-modulated softmax attention is itself positive semidefinite: the guarantee applies to the standalone kernel K. 4. First cheap experiment: train a 4-layer, 8-head Transformer on WikiText-2 or CIFAR-10 with matched parameter count and FLOPs. Compare ordinary attention, fixed alpha = 2, fixed alpha = 1, fixed alpha = 0.5, and the constrained learned-alpha model. Measure validation loss, gradient-norm spikes, attention entropy, and performance on sequences twice as long as those used in training. A successful result is faster loss descent or fewer instability events at equal compute, together with improved long-context extrapolation. Include an unconstrained alpha ablation to test whether alpha > 2 causes unstable or non-positive kernel behavior.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.