# Subresonant Power-Law Memory Bank

- ID: 331
- Canonical URL: https://synthcore.org/idea/331/subresonant-power-law-memory-bank
- API JSON: https://synthcore.org/api/idea/331.json
- API Markdown: https://synthcore.org/api/idea/331.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.01447](https://arxiv.org/abs/2607.01447)
- Category: architecture
- Solves: memory, sample-efficiency, scalability
- ML areas: ssm, rnn, transformer, world-model
- Math tags: dynamical-systems, harmonic-analysis, asymptotic-analysis, numerical-analysis
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Add a deterministic complex-valued state-space bank whose mode detunings become progressively smaller with mode index, Delta_n=c n^{-p}, while input couplings decay as B_n=b n^{-kappa}. For slowly varying or constant forcing, the summed state follows the paper's subresonant response and grows like t^{1-alpha}, providing controllable power-law memory with only O(N) recurrent state updates. This should improve long-context retention compared with a same-size unconstrained RNN or uniformly spaced SSM spectrum.

## Mathematical statement

The paper studies the accumulated response S(t)=sum_{n=1}^{infinity} B_n(1-exp(-i Delta_n t))/Delta_n, where t>=0 is time, Delta_n is the detuning of mode n, B_n is its forcing coefficient, and i is the imaginary unit. Under Delta_n~c n^{-p} with c!=0 and p>0, B_n~b n^{-kappa} with b!=0, and 0<alpha=(kappa-1)/p<1, the paper proves S(t) grows at order t^{1-alpha}; equivalently, a first-order perturbation with scale epsilon remains small only when epsilon t^{1-alpha}<<1. The neural adaptation uses N finite modes with angular detunings Delta_n=c n^{-p}, couplings B_n=b n^{-kappa}, and complex states z_n. For a piecewise-constant input q(t), the exact zero-order-hold update is z_n(t+h)=exp(-i Delta_n h)z_n(t)+B_n h phi_1(-i Delta_n h)q(t), where phi_1(x)=(exp(x)-1)/x; constant forcing therefore reproduces B_n(1-exp(-i Delta_n t))/Delta_n. Use conjugate-paired modes or separate real and imaginary states to obtain real outputs. The condition 0<(kappa-1)/p<1 gives the intended sublinear accumulation. Optional damping rho_n>=0 replaces exp(-i Delta_n h) by exp((-rho_n-i Delta_n)h) when bounded states are required.

## Key formulas

- $$\Delta_n\sim c n^{-p},\qquad B_n\sim b n^{-\kappa},\qquad c\neq 0,\ p>0.$$
- $$S(t)=\sum_{n=1}^{\infty}B_n\frac{1-\exp(-i\Delta_n t)}{\Delta_n},\qquad 0<\alpha=\frac{\kappa-1}{p}<1,$$ $$S(t)=O\!\left(t^{1-\alpha}\right).$$
- $$\varepsilon t^{1-\alpha}\ll 1,\qquad t\ll\varepsilon^{-1/(1-\alpha)}.$$
- $$z_n^{k+1}=e^{-i\Delta_n h}z_n^k+B_nh\,\phi_1(-i\Delta_n h)q_k,\qquad \phi_1(x)=\frac{e^x-1}{x},\quad y_k=\operatorname{Re}\!\left(\sum_{n=1}^{N}r_n z_n^k\right).$$

## Implementation notes

(1) Exact integration point: insert the bank as a residual memory block before the Transformer or MLP readout in a sequence model, or use it as the complete recurrent core of an SSM. At each token, project the input embedding x_k to a scalar or low-dimensional forcing q_k=W_in x_k, update all N modes, and add y_k=W_out Re(sum_n r_n z_n^k) to the main hidden state. Store each complex mode as two real channels; use conjugate pairs if the output projection is vector-valued. Initialize p and kappa so 0<(kappa-1)/p<1, for example p=1 and kappa=1.5. Set Delta_n=c n^{-p}, and choose c so that the smallest Delta_N is approximately 1/T_target for a target context length T_target. (2) Pseudocode: z=0; for each token k, q=W_in @ x[k]; for n=1..N, d=c*n**(-p), a=exp((-rho[n]-1j*d)*h), u=h*phi1((-rho[n]-1j*d)*h), z[n]=a*z[n]+B[n]*u*q; y[k]=W_out@real(sum(r[n]*z[n])). Add y[k] to the transformer stream. Compute phi1 with expm1(x)/x and use a Taylor branch phi1(x)=1+x/2+x^2/6 when |x|<1e-4. (3) The asymptotic exponent 1-alpha and the exact recurrence come from the paper; W_in, W_out, r, c, p, kappa, and damping are learned or initialized empirically. Estimate the observed memory exponent by fitting the slope of log(|y(t)|) against log(t) under constant q, and monitor gradient and state norms. (4) First cheap experiment: compare this bank against a vanilla diagonal SSM and GRU with equal recurrent state count on a synthetic delayed-copy task and the Long Range Arena retrieval task, using N=32 or 64 and contexts from 1k to 16k. Measure accuracy versus delay, validation perplexity, wall-clock cost, and retained signal after 10k steps. Success means higher delayed-retrieval accuracy or lower perplexity at equal parameters and FLOPs, with measured constant-input slope near 1-(kappa-1)/p and no exploding gradients. Ablate the power-law spectrum by replacing Delta_n with uniformly spaced frequencies while keeping parameter count fixed.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built an exact complex-valued zero-order-hold power-law memory bank with numerically stable phi1 evaluation, plus uniform-spectrum and GRU baselines and a delayed-copy benchmark. The math check passed: ZOH relative error was 1.6e-15 and the measured constant-input exponent was 0.463 versus the predicted 0.5. However, the proposed bank did not improve the main benchmark: power-law accuracy was 0.616 versus 0.675 for uniform spacing, so the claimed general win was not established.

### Mechanism check

- Verdict: Built an exact complex-valued zero-order-hold power-law memory bank with numerically stable phi1 evaluation, plus uniform-spectrum and GRU baselines and a delayed-copy benchmark. The math check passed: ZOH relative error was 1.6e-15 and the measured constant-input exponent was 0.463 versus the predicted 0.5. However, the proposed bank did not improve the main benchmark: power-law accuracy was 0.616 versus 0.675 for uniform spacing, so the claimed general win was not established.
- Confidence: 9/10
- Limitations: The benchmark used a small synthetic scalar delayed-copy task, short contexts up to 240 steps, fixed nonlearned bank coefficients, and CPU fallback after a shared-GPU cuDNN allocation failure. It did not test Long Range Arena, transformer residual integration, learned spectral parameters, larger contexts, or extensive seeds.

## Artifacts

- [report.md](https://synthcore.org/code/76/report.md)
- [results.json](https://synthcore.org/code/76/results.json)
- [run_experiment.py](https://synthcore.org/code/76/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/76)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.