# Capacity-aware compressive-plus-indexed memory

- ID: 894
- Canonical URL: https://synthcore.org/idea/894/capacity-aware-compressive-plus-indexed-memory
- API JSON: https://synthcore.org/api/idea/894.json
- API Markdown: https://synthcore.org/api/idea/894.md
- Verification status: unverified
- Source: [arXiv:2607.14144](https://arxiv.org/abs/2607.14144)
- Category: architecture
- Solves: accuracy, scalability, memory
- ML areas: ssm, attention, retrieval, inference-speedup
- Math tags: information-theory, probability, dynamical-systems
- Ratings: usefulness 6/10; difficulty 5/10; novelty 4/10

## Idea description

Replace a purely recurrent or state-space history summary with two explicitly separated paths: a fixed-size state channel for compressed sequence mixing and a query-dependent indexed channel for exact or near-exact retrieval. Train a lightweight gate to invoke top-k retrieval only when the recurrent state has insufficient evidence for the current query, preserving near-constant cost on ordinary tokens while preventing catastrophic failures on long-range exact-recall tasks.

## Mathematical statement

The paper assumes the Markov chain $X_{1:t}\rightarrow\Sigma_t\rightarrow\hat y_t$ conditioned on query $q_t$ and parameters $\theta$, meaning the prediction reads history $X_{1:t}$ only through the accessible state $\Sigma_t$. By the data-processing inequality, $I(X_{1:t};\hat y_t\mid q_t,\theta)\le I(X_{1:t};\Sigma_t\mid q_t,\theta)\le \mathcal H(\Sigma_t)\le B$, where $X_{1:t}$ is the history, $\hat y_t$ is the prediction, $q_t$ is the current query token, $\theta$ is model parameters independent of the history, $\Sigma_t$ is the data-bearing state exposed to the readout, $\mathcal H$ is Shannon entropy, and $B$ is the state-channel capacity in bits. A finite state therefore cannot expose more than $B$ history bits, even if its internal representation is high quality. The adaptation exposes a second indexed channel $R_t$ containing query-selected memory entries; with a state budget $B_s$ and $k$ retrieved entries quantized to $b$ bits each, the accessible information budget is approximately $B_s+kb$ rather than $B_s$, while the recurrent state remains fixed-size. The retrieval channel is not free: its computational cost is $O(kd)$ for an indexed lookup or $O(Ld)$ for dense attention over a length-$L$ context.

## Key formulas

- $$X_{1:t}\rightarrow\Sigma_t\rightarrow\hat{y}_t,\qquad I(X_{1:t};\hat{y}_t\mid q_t,\theta)\le I(X_{1:t};\Sigma_t\mid q_t,\theta)\le \mathcal{H}(\Sigma_t)\le B.$$
- $$\Sigma_t=f_\theta(\Sigma_{t-1},x_t),\qquad R_t=\operatorname{TopK}_{i\le t}\;s_\theta(q_t,m_i),\qquad h_t=\operatorname{Readout}_\theta(q_t,\Sigma_t,R_t).$$
- $$\mathcal{H}(\Sigma_t,R_t)\le B_s+kb,\qquad \operatorname{cost}(t)=O(1)+O(kd),$$
- $$g_t=\sigma\!\left(w^\top[\Sigma_t,q_t,c_t]+b_g\right),\qquad h_t=\operatorname{Readout}_\theta\!\left(q_t,\Sigma_t,g_tR_t\right).$$

## Implementation notes

Integrate this at the output interface of a small SSM, gated recurrent model, or linear-attention backbone. At token $t$, maintain a fixed-width state $\Sigma_t$ and update it with the backbone recurrence; store keys and values $(m_i,v_i)$ for a bounded or external memory. Compute a query key from $q_t$ and retrieve the top $k$ entries using an approximate-nearest-neighbor index, or use dense attention during the first prototype. Form a cheap confidence feature $c_t$, such as the entropy of the state-only output distribution, the margin between its top two logits, or the norm of a state/query compatibility vector. Compute $g_t=\operatorname{sigmoid}(w^\top[\Sigma_t,q_t,c_t])$. If $g_t$ is below a threshold, return the state-only prediction; otherwise attend over the retrieved values and combine them with the state through a two-layer readout. During training, use teacher-forced next-token loss plus a retrieval-budget penalty $\lambda\sum_t g_t$, and optionally force retrieval on examples whose label requires a known distant token. Pseudocode: `state=f(state,x[t]); confidence=conf(state,q[t]); gate=sigmoid(linear(state,q[t],confidence)); if gate>tau: R=TopK(index,query(q[t]),k); h=readout(q[t],state,R); else: h=readout(q[t],state); loss+=CE(h,y[t])+lambda*gate`. The paper supplies the information-capacity rationale; estimate $B_s$ empirically from state quantization entropy and estimate retrieval cost from measured ANN latency. First test on a synthetic Newton-apple-style stream or copy/retrieval benchmark with sequences of 4k-64k tokens, comparing an equal-width SSM, full attention, and the hybrid. Success means exact distant-retrieval accuracy approaching full attention, state-only failure increasing with context length, and at least 2-4x lower average retrieval/FLOP cost than always-on attention at matched accuracy.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.