Subresonant Power-Law Memory Bank
Implementation & benchmark of arXiv:2607.01447 — Small Denominators and Subresonant Accumulation in Weakly Nonlinear Dispersive Dynamics
Source paper: Small Denominators and Subresonant Accumulation in Weakly Nonlinear Dispersive Dynamics arXiv:2607.01447 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Add a deterministic complex-valued state-space bank whose mode detunings become progressively smaller with mode index, Delta_n=c n^{-p}, while input couplings decay as B_n=b n^{-kappa}. For slowly varying or constant forcing, the summed state follows the paper's subresonant response and grows like t^{1-alpha}, providing controllable power-law memory with only O(N) recurrent state updates. This should improve long-context retention compared with a same-size unconstrained RNN or uniformly spaced SSM spectrum.
Formulas
Mathematical statement
The paper studies the accumulated response S(t)=sum_{n=1}^{infinity} B_n(1-exp(-i Delta_n t))/Delta_n, where t>=0 is time, Delta_n is the detuning of mode n, B_n is its forcing coefficient, and i is the imaginary unit. Under Delta_n~c n^{-p} with c!=0 and p>0, B_n~b n^{-kappa} with b!=0, and 0<alpha=(kappa-1)/p<1, the paper proves S(t) grows at order t^{1-alpha}; equivalently, a first-order perturbation with scale epsilon remains small only when epsilon t^{1-alpha}<<1. The neural adaptation uses N finite modes with angular detunings Delta_n=c n^{-p}, couplings B_n=b n^{-kappa}, and complex states z_n. For a piecewise-constant input q(t), the exact zero-order-hold update is z_n(t+h)=exp(-i Delta_n h)z_n(t)+B_n h phi_1(-i Delta_n h)q(t), where phi_1(x)=(exp(x)-1)/x; constant forcing therefore reproduces B_n(1-exp(-i Delta_n t))/Delta_n. Use conjugate-paired modes or separate real and imaginary states to obtain real outputs. The condition 0<(kappa-1)/p<1 gives the intended sublinear accumulation. Optional damping rho_n>=0 replaces exp(-i Delta_n h) by exp((-rho_n-i Delta_n)h) when bounded states are required.
Implementation notes
(1) Exact integration point: insert the bank as a residual memory block before the Transformer or MLP readout in a sequence model, or use it as the complete recurrent core of an SSM. At each token, project the input embedding x_k to a scalar or low-dimensional forcing q_k=W_in x_k, update all N modes, and add y_k=W_out Re(sum_n r_n z_n^k) to the main hidden state. Store each complex mode as two real channels; use conjugate pairs if the output projection is vector-valued. Initialize p and kappa so 0<(kappa-1)/p<1, for example p=1 and kappa=1.5. Set Delta_n=c n^{-p}, and choose c so that the smallest Delta_N is approximately 1/T_target for a target context length T_target. (2) Pseudocode: z=0; for each token k, q=W_in @ x[k]; for n=1..N, d=c*n**(-p), a=exp((-rho[n]-1j*d)*h), u=h*phi1((-rho[n]-1j*d)*h), z[n]=a*z[n]+B[n]*u*q; y[k]=W_out@real(sum(r[n]*z[n])). Add y[k] to the transformer stream. Compute phi1 with expm1(x)/x and use a Taylor branch phi1(x)=1+x/2+x^2/6 when |x|<1e-4. (3) The asymptotic exponent 1-alpha and the exact recurrence come from the paper; W_in, W_out, r, c, p, kappa, and damping are learned or initialized empirically. Estimate the observed memory exponent by fitting the slope of log(|y(t)|) against log(t) under constant q, and monitor gradient and state norms. (4) First cheap experiment: compare this bank against a vanilla diagonal SSM and GRU with equal recurrent state count on a synthetic delayed-copy task and the Long Range Arena retrieval task, using N=32 or 64 and contexts from 1k to 16k. Measure accuracy versus delay, validation perplexity, wall-clock cost, and retained signal after 10k steps. Success means higher delayed-retrieval accuracy or lower perplexity at equal parameters and FLOPs, with measured constant-input slope near 1-(kappa-1)/p and no exploding gradients. Ablate the power-law spectrum by replacing Delta_n with uniformly spaced frequencies while keeping parameter count fixed.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built an exact complex-valued zero-order-hold power-law memory bank with numerically stable phi1 evaluation, plus uniform-spectrum and GRU baselines and a delayed-copy benchmark. The math check passed: ZOH relative error was 1.6e-15 and the measured constant-input exponent was 0.463 versus the predicted 0.5. However, the proposed bank did not improve the main benchmark: power-law accuracy was 0.616 versus 0.675 for uniform spacing, so the claimed general win was not established.
- Agent confidence
- 9/10
- Baseline
- Uniform spectrum: MSE 0.5158, accuracy 0.6754, runtime 2.66 s; GRU: MSE 0.6248, accuracy 0.4141, runtime 18.11 s.
- Idea
- Power-law spectrum: MSE 0.5547, accuracy 0.6158, runtime 2.79 s; measured slope 0.4632 versus predicted 0.5; ZOH relative error 1.61e-15.
Limitations: The benchmark used a small synthetic scalar delayed-copy task, short contexts up to 240 steps, fixed nonlearned bank coefficients, and CPU fallback after a shared-GPU cuDNN allocation failure. It did not test Long Range Arena, transformer residual integration, learned spectral parameters, larger contexts, or extensive seeds.
How to run: python3 run_experiment.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Artifacts
- report.md 1.4 KB View
- results.json 1.2 KB View Raw
- run_experiment.py 5.9 KB View Raw