SBP Energy-Stable Sequence Mixer

(1) Integration point: replace one dense token-mixing block in a small Transformer, MLP-Mixer, or state-space model. Let H have shape batch x sequence_length x hidden_width. Construct P=diag(w_1,...,w_N) with positive weights and a derivative matrix D. For the first MVP, use a fixed SBP matrix for N=32 or 64 sequence positions, computed from a polynomial basis; a second version can use the paper's exponential basis or interpolate arbitrary token sequences onto the nonuniform Gaussian nodes. Apply the derivative independently to each hidden channel.
(2) Pseudocode:
P = diag(softplus(p) + eps)
S = A - transpose(A)
Q = S + 0.5 * B
D = inverse(P) @ Q
Z = -a * einsum("ij,bjd->bid", D, H)
Y = activation(Z @ W + H @ U)
H_next = H + gamma * Y
Estimate normP with power iteration on sqrt(P) @ D @ inverse_sqrt(P), then set gamma=min(gamma_max, 0.9/(abs(a)*normP)). Here A is an unconstrained learnable matrix, S is skew-symmetric, B is fixed, and a is either learned or initialized to one. For causal sequences choose B=diag(-1,0,...,0,1) and optionally add a penalty on the terminal boundary energy. For periodic sequences use B=0 and a cyclic skew structure.
(3) Computed directly from the mathematics: the SBP parameterization, the weighted energy identity, and the P-norm stability scale. Estimated empirically: the spectral norm, the transport speed a, and whether learning S improves task loss. The operator costs O(N^2 d) for a dense D, but a banded SBP matrix can reduce this to O(N d), and Gaussian nodes may permit a smaller effective N after pooling or interpolation.
(4) First cheap experiment: train a six-layer sequence classifier and an autoregressive character model on copy/reversal tasks and a small WikiText-2 split. Compare a standard residual token mixer against the SBP mixer at equal hidden size and approximately equal FLOPs. Record activation norms, gradient norms, loss spikes, validation accuracy, and perplexity versus depth. The primary falsifiable target is fewer optimization instabilities and equal or better perplexity; a stronger result is stable training at twice the baseline depth or learning rate, with a banded implementation approaching linear rather than quadratic sequence mixing cost.