# Bounded Exponential Anchor Memory

- ID: 169
- Canonical URL: https://synthcore.org/idea/169/bounded-exponential-anchor-memory
- API JSON: https://synthcore.org/api/idea/169.json
- API Markdown: https://synthcore.org/api/idea/169.md
- Verification status: unverified
- Source: [arXiv:2608.25298](https://arxiv.org/abs/2608.25298)
- Category: memory
- Solves: memory, stability, scalability
- ML areas: transformer, ssm, kv-cache, inference-speedup
- Math tags: geometry, probability, dynamical-systems
- Ratings: usefulness 6/10; difficulty 4/10; novelty 4/10

## Idea description

Replace a growing stream of cached token or event representations with one normalized exponentially updated anchor. Use the anchor as a global context vector or as a compressed key/value memory; the state retains recent information while avoiding the full movement cost of repeatedly attaching to the previous token.

## Mathematical statement

The paper defines points p_0,p_1,...,p_N in the Euclidean unit ball B_2^d(1), and the state recursion x_0=p_0 and x_i=gamma x_{i-1}+(1-gamma)p_i, where gamma is in [0,1]. Expanding the recursion gives x_i=gamma^i p_0+(1-gamma) sum_{j=1}^i gamma^{i-j}p_j, so x_i is a convex combination of observed points and remains in B_2^d(1). The insertion cost is C_N=sum_{i=1}^N ||p_i-x_{i-1}||^alpha, where alpha>0. For arbitrary input sequences and 0<alpha<=3, the paper states that the largest asymptotic mean cost is (2/(1+gamma))^alpha, strictly below the path value 2^alpha when gamma>0. For independent uniform points, the minimizing constant parameter obeys 1-gamma_N^*=Theta(N^{-1/2}); for alpha=2, gamma_N^*=1-N^{-1/2}+1/(2N)+O(N^{-3/2}). In the neural adaptation, p_i is a normalized token embedding and x_i is the compressed memory vector.

## Key formulas

- $$x_0=p_0,\qquad x_i=\gamma x_{i-1}+(1-\gamma)p_i,\qquad 0\leq\gamma\leq1.$$
- $$x_i=\gamma^i p_0+(1-\gamma)\sum_{j=1}^{i}\gamma^{i-j}p_j.$$
- $$C_N=\sum_{i=1}^{N}\left\|p_i-x_{i-1}\right\|^{\alpha},\qquad \limsup_{N\to\infty}\frac{C_N}{N}\leq\left(\frac{2}{1+\gamma}\right)^{\alpha}\quad(0<\alpha\leq3).$$
- $$\gamma_N^*=1-N^{-1/2}+\frac{1}{2}N^{-1}+O(N^{-3/2})\qquad(\alpha=2).$$

## Implementation notes

Integrate the method into a streaming Transformer at the layer that produces per-token representations. Project each representation h_t into a memory dimension d and normalize it as p_t=h_t/(||h_t||_2+epsilon), so the unit-ball assumption is approximately satisfied. Maintain one state vector x_t per sequence, layer, or attention head. Before processing token t, use x_{t-1} as a global context token, or create one compressed key/value pair k_t=W_k x_{t-1} and v_t=W_v x_{t-1}. After the token is processed, update the state with x_t=gamma x_{t-1}+(1-gamma)p_t. Pseudocode is: initialize x=p_0; for each token, compute p=normalize(project(h)); compute context=f(x); compute the network output using h and context; then set x=gamma*x+(1-gamma)*p. For a context horizon N, initialize gamma=1-1/sqrt(N)+0.5/N using the paper's alpha=2 expansion, and compare this with gamma values 0.9, 0.99, and 0.999. The exact mathematical computations are the recurrence, geometric weights, and worst-case proxy B(gamma,alpha)=(2/(1+gamma))^alpha. Estimate empirically the embedding radius, average attachment cost, perplexity, and gradient variance. Start with a small causal Transformer on WikiText-2 or enwik8, comparing full KV caching, last-token memory, mean pooling, and this one-vector memory. Measure peak KV memory, tokens per second, validation perplexity versus context length, and worst-decile attachment cost. Success requires a large memory reduction with less perplexity degradation than last-token compression at equal parameter count.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.