# Tempered Log-Memory State Mixer

- ID: 1682
- Canonical URL: https://synthcore.org/idea/1682/tempered-log-memory-state-mixer
- API JSON: https://synthcore.org/api/idea/1682.json
- API Markdown: https://synthcore.org/api/idea/1682.md
- Verification status: unverified
- Source: [arXiv:2608.01674](https://arxiv.org/abs/2608.01674)
- Category: architecture
- Solves: accuracy, stability, scalability
- ML areas: ssm, transformer, attention
- Math tags: stochastic-processes, probability, harmonic-analysis
- Ratings: usefulness 5/10; difficulty 5/10; novelty 5/10

## Idea description

Replace or augment an exponential state-space memory branch with a causal convolution whose lag-j weight is exp(-lambda j) ell(j)/j. The 1/j boundary provides broad logarithmic memory, while lambda supplies an explicit finite memory scale and prevents uncontrolled accumulation from an untempered long-memory kernel.

## Mathematical statement

The paper studies X_{N,n} = sum_{j=1}^{infinity} exp(-lambda_N j) ell(j)/j epsilon_{n-j}, where epsilon_k are i.i.d. innovations, lambda_N > 0 is a tempering parameter, and ell is slowly varying, meaning ell(cx)/ell(x) tends to 1 for every fixed c > 0. For context length N, the cumulative kernel mass is Q_N = sum_{j=1}^N exp(-lambda_N j) ell(j)/j, while the untempered mass is L(N) = sum_{j=1}^N ell(j)/j. The theorem gives Q_N asymptotic to L(N) when N lambda_N = O(1), and Q_N asymptotic to L(1/lambda_N) when N lambda_N tends to infinity. The neural adaptation sets ell(j) = 1, so L(N) is the harmonic sum H_N, and normalizes the finite-context mixer by Q_T(lambda) = sum_{j=1}^T exp(-lambda j)/j.

## Key formulas

- $$X_{N,n}=\sum_{j=1}^{\infty}e^{-\lambda_N j}\frac{\ell(j)}{j}\varepsilon_{n-j}.$$
- $$Q_N=\sum_{j=1}^{N}e^{-\lambda_N j}\frac{\ell(j)}{j},\qquad L(N)=\sum_{j=1}^{N}\frac{\ell(j)}{j}.$$
- $$Q_N\sim L(N)\quad\text{if }N\lambda_N=O(1),\qquad Q_N\sim L(1/\lambda_N)\quad\text{if }N\lambda_N\to\infty.$$
- $$m_t=\frac{1}{Q_T(\lambda)}\sum_{j=1}^{\min(T,t-1)}\frac{e^{-\lambda j}}{j}v_{t-j},\qquad Q_T(\lambda)=\sum_{j=1}^{T}\frac{e^{-\lambda j}}{j}.$$

## Implementation notes

Integrate this as a depthwise causal memory branch in a Transformer block or as a replacement for one diagonal SSM branch. Given hidden states h_1 through h_T in R^d, project v_t = W_v h_t, optionally compute a gate g_t = sigmoid(W_g h_t), and compute the causal tempered-log convolution m_t. Use ell(j) = 1 initially. Parameterize lambda as softplus(rho) + 10^{-4}, either once per layer or once per channel group. Compute Q exactly for the current maximum context length so output scale is independent of T. Pseudocode: `Q = sum(exp(-lambda*j)/j for j=1..T); for t in 1..T: m[t] = sum(exp(-lambda*j)/j * v[t-j] / Q for j=1..min(T,t-1)); y[t] = h[t] + W_o(g[t] * m[t])`. Direct convolution costs O(T^2 d), so first use FFT convolution for long sequences or approximate the kernel with K logarithmically spaced lag bins. For bin k covering lags [2^k, 2^(k+1)), store a prefix-sum difference of v and multiply it by the average kernel weight in that bin, reducing the branch to approximately O(KTd). The paper supplies the kernel and asymptotic normalization; lambda is estimated by validation or learned from a log-uniform initialization spanning inverse memory scales from 16 to the sequence length. Test first on WikiText-103 or Long Range Arena against an equal-parameter Transformer with exponential SSM memory. Evaluate perplexity or sequence accuracy at lengths 512, 2048, and 8192, plus activation RMS and gradient norms. Success means lower long-context loss without length-dependent activation growth or instability.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.