Sequence-Distortion Hidden-State Regularizer

(1) Integrate this at the final hidden-state tensor of a Transformer, RNN, or SSM, before the task head. Given h in R^{B x T x k}, sample P valid position pairs (b,i,j), compute d_ij=||h[b,i]-h[b,j]||_2, and add lambda_seq L_seq to the ordinary language-model or classification loss. Exclude padding and, when appropriate, avoid pairs crossing document boundaries. (2) Pseudocode: choose alpha in (1/k,1], C>=1, and lambda_seq. For each batch sample M pairs with separations drawn approximately log-uniformly from 1 through T-1. Set r=abs(i-j), lower=(1/C)*((max(0,floor(r/C-C))+1)^alpha), upper=C*((C*r+C+1)^alpha)+C, and d=sqrt(sum((h_i-h_j)^2)+1e-8). Accumulate relu(lower-d)^2+relu(d-upper)^2, divide by M, and add the result to the task loss. A smoother alternative uses softplus(beta*(lower-d))^2 and softplus(beta*(d-upper))^2. (3) The paper supplies the coarse admissibility condition and distortion inequalities; the implementation estimates only Euclidean pair distances. Treat C, alpha, and lambda_seq as hyperparameters. Start with C=2 or 4, lambda_seq in [1e-3,1e-1], and normalize h by batch RMS before distance computation if the auxiliary loss causes scale drift. Anneal lambda_seq from zero during the first 5-10% of training. (4) First experiment: train a small 6-layer causal Transformer on WikiText-2 or TinyStories with hidden widths k in {64,128,256}, comparing the baseline against alpha in {0.25,0.5,0.75,1.0}. Measure validation perplexity, long-context perplexity, gradient-norm variance, and median d_ij versus log separation. The key falsifiable signal is reduced long-range hidden-state collapse and improved perplexity at context lengths longer than those seen during training, without a material increase in training loss. Use M=128-512 pairs per batch and benchmark vectorized pair-distance overhead.