Markov-Increment Window Encoder

Integrate the method before a transformer attention block, operating on local window embeddings rather than raw tokens. Choose window lengths m=1,...,k, with k=4 or 8. Let phi_m(t) be a d-dimensional embedding of tokens X_t,...,X_{t+m-1}, produced by a shared-weight MLP, depthwise one-dimensional convolution, or small local attention module. For each m>1, train predictors g_{m,r} from shorter windows X_{t:t+r-1} to phi_m(t), for every r<m, and form the residual e_m(t)=phi_m(t)-sum_{r<m} alpha_{m,r}g_{m,r}(X_{t:t+r-1}); implement the sum using concatenation followed by a linear projection. This approximates projection onto the incremental component. Stack e_1(t),...,e_k(t) into z(t). Estimate a one-step feature transition operator K by ridge regression, K=argmin_A sum_t ||u_{t+1}-Au_t||^2+lambda||A||_F^2, where u_t is a low-dimensional projected token feature. Apply the displayed tridiagonal precision transform to the length-indexed feature stack: q=(I-K^2)^(-1)Bz, where B has endpoint diagonal I, interior diagonal I+K^2, and adjacent off-diagonal -K. To avoid an expensive matrix square root, begin with this precision-style preconditioner rather than explicitly computing T_k(K)^(-1/2). Maintain K and the shorter-window predictors with exponential moving averages and stop gradients through them during the first experiment. The paper determines the Toeplitz structure and tridiagonal coefficients; K, predictor parameters, ridge coefficient, and residual dimensions are estimated empirically. Test first on WikiText-103 or a synthetic reversible Markov token generator with a 2-layer, 256-dimensional transformer. Compare against standard overlapping convolutional/window embeddings at equal parameter count and FLOPs. Measure validation perplexity, loss descent speed, gradient covariance condition number, attention entropy, and feature cross-covariance. A successful result is lower perplexity or faster loss descent at equal compute, at least a 20% reduction in cross-window feature covariance, and no increase in activation memory.