Power-Law Volterra Memory

Integrate the idea at the residual-stream level of a causal Transformer, SSM, or RNN. Given hidden states x[0:L,d], project them with V∈R^{d×d_m}, compute a causal Volterra memory m[i] from earlier projected states, and add a gated residual y[i]=x[i]+g[i]⊙m[i], where g[i]=sigmoid(linear(x[i])) and ε=10^{-6}. Use t_i=(i+1)/L, initialize α=0, β=0, and γ=−0.75; parameterize α=−0.5+softplus(a), γ=−1+0.5·sigmoid(c) so that α>−1/2 and −1<γ<−1/2, and set β=H−α−γ−3/2 with H=softplus(h), or initially keep all exponents fixed. Precompute K[i,j] on the maximum context length using numerical quadrature, a closed-form incomplete-beta implementation, or a 32-bin logarithmic-lag approximation. For an efficient MVP, approximate the integral by midpoint quadrature with 4–8 points per interval and truncate to a power-of-two window; alternatively approximate the resulting lag weights by a sum of 8 exponentials and evaluate the memory with eight recurrent states, reducing O(L²) cost to O(8L). Pseudocode is: compute t; for i in 1..L, for j<i evaluate K[i,j], row-normalize to Ktilde, accumulate m[i]=sum_j Ktilde[i,j]@(x[j]@V), then y[i]=x[i]+sigmoid(Gx[i])⊙(m[i]@O). The paper determines the admissible exponent region and interprets γ+3/2 as roughness and H as scale; quadrature error, truncation length, and whether learned exponents improve validation loss must be measured empirically. First experiment: replace one recurrent layer or one Transformer long-context block in a 125M-parameter language model and train on WikiText-103 or PG19 with 2k–8k contexts. Compare against a same-parameter dilated convolution and a diagonal SSM at equal FLOPs. Success means lower perplexity at long contexts, better loss on sequences longer than training length, or equal perplexity with fewer memory states; also report wall-clock cost and gradient-norm stability as γ approaches −1/2.