# Stable-charge attention kernel

- ID: 2515
- Canonical URL: https://synthcore.org/idea/2515/stable-charge-attention-kernel
- API JSON: https://synthcore.org/api/idea/2515.json
- API Markdown: https://synthcore.org/api/idea/2515.md
- Verification status: unverified
- Source: [arXiv:2608.18335](https://arxiv.org/abs/2608.18335)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: attention, transformer, graph-nn
- Math tags: probability, stochastic-processes, functional-analysis, metric-geometry
- Ratings: usefulness 6/10; difficulty 4/10; novelty 5/10

## Idea description

Attach each token or graph node a learned scalar charge q_i and add a fractional stable kernel K_ij = exp(-tau D |q_i-q_j|^alpha) to the interaction mechanism. Constrain 0 < alpha <= 2, the exact range in which the kernel is positive semidefinite for arbitrary finite real charge sets, and optionally make tau layer-dependent to obtain multiscale interactions. This provides a principled alternative to unconstrained learned distance biases and can be used either as an attention-logit bias or as a separate normalized kernel-mixing branch.

## Mathematical statement

The paper's scale-covariant symmetric Levy lift satisfies E[exp(i xi (L_u-L_0))] = exp(-D u |xi|^alpha), where D >= 0 is the noise scale, u >= 0 is operational time, xi is a real frequency, and 0 < alpha <= 2. Averaging the random unitary exp(i L_u Q) produces decay between charge sectors q_a and q_b at rate Gamma_ab = D |q_a-q_b|^alpha. The induced pairwise kernel is K_u(q_a,q_b) = exp[-D u |q_a-q_b|^alpha]. Schoenberg's theorem gives that K_ij = exp[-tau D |q_i-q_j|^alpha] is positive semidefinite for every finite set of real charges q_1,...,q_n exactly when 0 < alpha <= 2; for alpha > 2, three equally spaced charges provide an analytic obstruction. The transferred asset is therefore a globally valid positive-definite affinity with a learnable fractional exponent. The exponential also gives the entrywise semigroup identity K_ij(tau_1+tau_2) = K_ij(tau_1) K_ij(tau_2).

## Key formulas

- $$\mathbb{E}\!\left[e^{i\xi(L_u-L_0)}\right]=e^{-Du|\xi|^{\alpha}},\qquad 0<\alpha\le 2.$$
- $$\Gamma_{ab}=D|q_a-q_b|^{\alpha}.$$
- $$K_{ij}(\tau)=\exp\!\left[-\tau D|q_i-q_j|^{\alpha}\right],\qquad K(\tau)\succeq 0\ \text{for all finite }\{q_i\}\subset\mathbb{R}\text{ when }0<\alpha\le2.$$
- $$K_{ij}(\tau_1+\tau_2)=K_{ij}(\tau_1)K_{ij}(\tau_2),\qquad K(\tau_1+\tau_2)=K(\tau_1)\odot K(\tau_2).$$

## Implementation notes

1. Integration point: add a stable-charge interaction branch to a Transformer self-attention block before value aggregation. For hidden states H of shape batch x sequence x d, compute one scalar charge per token, q_i = w_q^T h_i + b_q. Store unconstrained parameters rho and a, and define D = softplus(rho) + epsilon and alpha = epsilon + (2 - 2 epsilon) sigmoid(a), with epsilon = 1e-3. Let tau = softplus(tau_raw) be a learned nonnegative layer scale. 2. Pseudocode: `q = charge_linear(H).squeeze(-1)`; `dist = abs(q[:, :, None] - q[:, None, :])`; `stable_bias = -tau * D * dist.pow(alpha)`; `logits = Q @ K.transpose(-1,-2) / sqrt(d_head) + stable_bias`; `A = masked_softmax(logits)`; `Y = A @ V`. Also test a separate normalized kernel branch: `W = exp(stable_bias)` and `Y_stable = (W @ V) / (W.sum(-1, keepdim=True) + eps)`, followed by a learned gate combining Y and Y_stable. Compute distances and powers in fp32 before casting to the attention dtype. 3. The paper supplies the exponent range, positive-semidefinite kernel guarantee, and entrywise semigroup law. Backpropagation estimates q_i, D, alpha, and tau; no eigenvalue computation is needed. Do not claim that content-modulated softmax attention is itself positive semidefinite: the guarantee applies to the standalone kernel K. 4. First cheap experiment: train a 4-layer, 8-head Transformer on WikiText-2 or CIFAR-10 with matched parameter count and FLOPs. Compare ordinary attention, fixed alpha = 2, fixed alpha = 1, fixed alpha = 0.5, and the constrained learned-alpha model. Measure validation loss, gradient-norm spikes, attention entropy, and performance on sequences twice as long as those used in training. A successful result is faster loss descent or fewer instability events at equal compute, together with improved long-context extrapolation. Include an unconstrained alpha ablation to test whether alpha > 2 causes unstable or non-positive kernel behavior.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.