Log-Rank Fluctuation Channels

(1) Integration point: insert this module before a causal self-attention block, or replace every second attention block in a small causal Transformer. The input is the sequence of hidden states h[1:N], each in R^d. Estimate the centering vector mu with an exponential moving average during training; for strict prefix causality, instead use the current prefix mean. Fix a maximum context N so the log-rank weights are deterministic and future-independent. (2) Pseudocode: precompute w[i,n]=(log(N/i))^n/n! for i=1,...,N and n=0,...,K; initialize S[n]=0 for all n; for t=1,...,N, update S[n] = S[n] + w[t,n]*(h[t]-mu)/sqrt(N) for every n; concatenate S[0],...,S[K]; apply LayerNorm and a bottleneck MLP phi; project with W_o and add the result to h[t]. This is the adapted paper formula evaluated on each causal prefix. (3) Compute log-rank weights analytically. Learn the MLP and projection parameters by backpropagation; estimate mu and each channel's RMS online. Initialize W_o or a scalar output gate near zero so optimization begins close to the baseline. Test K in {0,1,2,4}, with optional RMS normalization per channel. (4) First cheap experiment: train a 6-layer, 256-dimensional causal Transformer on WikiText-103 with context length 1024. Compare the baseline, a baseline with an ordinary cumulative-mean channel, and the log-rank hierarchy at matched parameter count and FLOPs. Measure validation perplexity, perplexity by token distance, loss versus wall-clock time, and streaming memory. Success means lower long-range perplexity at equal compute, or matching baseline perplexity after removing one attention layer; the module itself should require only O(Kd) streaming state at inference.