Unverified 2026

Tempered Log-Memory State Mixer

Implementation & benchmark of arXiv:2608.01674 — Limit Theorems for Tempered Linear Processes with Innovations in the Domain of Attraction of a Stable Law

Usefulness5/10

Difficulty5/10

Novelty5/10

Architecture Accuracy Stability Scalability Ssm Transformer Attention Stochastic processes Probability Harmonic analysis

Source paper: Limit Theorems for Tempered Linear Processes with Innovations in the Domain of Attraction of a Stable Law arXiv:2608.01674 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace or augment an exponential state-space memory branch with a causal convolution whose lag-j weight is exp(-lambda j) ell(j)/j. The 1/j boundary provides broad logarithmic memory, while lambda supplies an explicit finite memory scale and prevents uncontrolled accumulation from an untempered long-memory kernel.

Formulas

$$X_{N,n}=\sum_{j=1}^{\infty}e^{-\lambda_N j}\frac{\ell(j)}{j}\varepsilon_{n-j}.$$

$$Q_N=\sum_{j=1}^{N}e^{-\lambda_N j}\frac{\ell(j)}{j},\qquad L(N)=\sum_{j=1}^{N}\frac{\ell(j)}{j}.$$

$$Q_N\sim L(N)\quad\text{if }N\lambda_N=O(1),\qquad Q_N\sim L(1/\lambda_N)\quad\text{if }N\lambda_N\to\infty.$$

$$m_t=\frac{1}{Q_T(\lambda)}\sum_{j=1}^{\min(T,t-1)}\frac{e^{-\lambda j}}{j}v_{t-j},\qquad Q_T(\lambda)=\sum_{j=1}^{T}\frac{e^{-\lambda j}}{j}.$$

Mathematical statement

The paper studies X_{N,n} = sum_{j=1}^{infinity} exp(-lambda_N j) ell(j)/j epsilon_{n-j}, where epsilon_k are i.i.d. innovations, lambda_N > 0 is a tempering parameter, and ell is slowly varying, meaning ell(cx)/ell(x) tends to 1 for every fixed c > 0. For context length N, the cumulative kernel mass is Q_N = sum_{j=1}^N exp(-lambda_N j) ell(j)/j, while the untempered mass is L(N) = sum_{j=1}^N ell(j)/j. The theorem gives Q_N asymptotic to L(N) when N lambda_N = O(1), and Q_N asymptotic to L(1/lambda_N) when N lambda_N tends to infinity. The neural adaptation sets ell(j) = 1, so L(N) is the harmonic sum H_N, and normalizes the finite-context mixer by Q_T(lambda) = sum_{j=1}^T exp(-lambda j)/j.

Implementation notes

Integrate this as a depthwise causal memory branch in a Transformer block or as a replacement for one diagonal SSM branch. Given hidden states h_1 through h_T in R^d, project v_t = W_v h_t, optionally compute a gate g_t = sigmoid(W_g h_t), and compute the causal tempered-log convolution m_t. Use ell(j) = 1 initially. Parameterize lambda as softplus(rho) + 10^{-4}, either once per layer or once per channel group. Compute Q exactly for the current maximum context length so output scale is independent of T. Pseudocode: Q = sum(exp(-lambda*j)/j for j=1..T); for t in 1..T: m[t] = sum(exp(-lambda*j)/j * v[t-j] / Q for j=1..min(T,t-1)); y[t] = h[t] + W_o(g[t] * m[t]). Direct convolution costs O(T^2 d), so first use FFT convolution for long sequences or approximate the kernel with K logarithmically spaced lag bins. For bin k covering lags [2^k, 2^(k+1)), store a prefix-sum difference of v and multiply it by the average kernel weight in that bin, reducing the branch to approximately O(KTd). The paper supplies the kernel and asymptotic normalization; lambda is estimated by validation or learned from a log-uniform initialization spanning inverse memory scales from 16 to the sequence length. Test first on WikiText-103 or Long Range Arena against an equal-parameter Transformer with exponential SSM memory. Evaluate perplexity or sequence accuracy at lengths 512, 2048, and 8192, plus activation RMS and gradient norms. Success means lower long-context loss without length-dependent activation growth or instability.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.