✗ Failed on benchmark 2026

Prony Memory Bank for Linear-Time Sequence Modeling

Implementation & benchmark of arXiv:2607.05151 — On data-driven parameterizations of multidimensional generalized Langevin dynamics in the presence of a quadratic potential

Usefulness7/10

Difficulty5/10

Novelty4/10

Architecture Speedup Memory Stability Accuracy Attention Ssm Inference speedup Memory Stochastic processes Dynamical systems Linear algebra Spectral theory

Source paper: On data-driven parameterizations of multidimensional generalized Langevin dynamics in the presence of a quadratic potential arXiv:2607.05151 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace quadratic self-attention over a sequence with a bank of K auxiliary exponentially decaying states whose rates are fitted directly from the empirical autocorrelation of the sequence features. Each mode captures a distinct time scale, so the module can represent short- and long-range dependencies with O(TK) computation and O(K) recurrent memory rather than storing all previous tokens. Constrain decay rates to be positive and use the paper's extended Markovian block structure to obtain a stable recurrent realization.

Formulas

$$\mathrm{d}\begin{bmatrix}Y\\ Z\\ X\end{bmatrix}=\begin{bmatrix}0&B^{T}&-\Lambda_{0}/m\\-C&\Delta&0\\I&0&0\end{bmatrix}\begin{bmatrix}Y\\ Z\\ X\end{bmatrix}\,\mathrm{d}t+\begin{bmatrix}0\\L\\0\end{bmatrix}\,\mathrm{d}W.$$

$$\Sigma_{0}=\frac{1}{\beta m}\begin{bmatrix}I&0&0\\0&\Sigma_{22}&0\\0&0&m\Lambda_{0}^{-1}\end{bmatrix},\qquad \Lambda_{0}=mB_{1}^{T}C_{1},\qquad L_{2}=-C_{1}^{-1}L_{1}=0.$$

$$R(\tau)\approx\sum_{k=1}^{K}A_{k}e^{-\lambda_{k}\tau},\qquad \lambda_k>0,$$

$$s_{k,t+1}=\rho_k s_{k,t}+G_k u_t,\quad \rho_k=e^{-\lambda_k\Delta t},\quad y_t=\sum_{k=1}^{K}H_k s_{k,t}.$$

Mathematical statement

Equation (22)/(35) constructs an extended Markov process with velocity-like variable Y in R^d, auxiliary state Z in R^(N-2d), and position-like variable X in R^d: d[Y;Z;X] = [[0,B^T,-Lambda_0/m],[-C,Delta,0],[I,0,0]][Y;Z;X]dt + [0;L;0]dW. Here B and C couple the observable and auxiliary states, Delta controls auxiliary-state decay, Lambda_0 is the effective stiffness matrix, m is a positive scalar mass, L is the noise factor, and W is Brownian motion. Proposition 7 states that Lambda_0 is symmetric positive definite, L_2=0, and the stationary covariance has the block form Sigma_0=(beta m)^(-1) diag(I,Sigma_22,m Lambda_0^(-1)); this motivates enforcing positive covariance and stable decay in the neural realization. The paper's data-driven principle is to approximate an observed autocorrelation matrix R(tau) by a finite Prony series R(tau) approximately equal to sum_{k=1}^K A_k exp(-lambda_k tau), where lambda_k>0 are decay rates and A_k are matrix amplitudes. For neural inputs u_t, use the adapted discrete Markov realization s_{k,t+1}=rho_k s_{k,t}+G_k u_t with rho_k=exp(-lambda_k Delta t), and output y_t=sum_k H_k s_{k,t}; positive lambda_k gives |rho_k|<1 and therefore bounded memory.

Implementation notes

Integrate this as a causal replacement for a self-attention block in a transformer encoder, speech model, or time-series model. Let u_t in R^d be the normalized token embedding after a small input projection. Maintain K auxiliary states s_k in R^r, with K between 4 and 32 and r=d/K or a smaller bottleneck. First collect a calibration stream of hidden features from a baseline model and estimate the lag autocorrelation R_hat(j)=1/(T-j) sum_{t=1}^{T-j} u_t u_{t+j}^T for j=0,...,J. Fit the Prony model R_hat(j) approximately sum_k A_k rho_k^j by nonlinear least squares, parameterizing rho_k=sigmoid(a_k) so 0<rho_k<1; initialize the rates on a logarithmic grid and optionally alternate least-squares updates for A_k with gradient updates for a_k. Factor each amplitude A_k approximately H_k G_k, using an eigendecomposition or rank-r SVD, and then fine-tune all projections end-to-end. At every token, run the recurrence s_k = rho_k*s_k + G_k*u_t and y_t = LayerNorm(u_t + sum_k H_k*s_k); feed y_t to the usual MLP. A more faithful block-structured variant can concatenate all s_k into Z and use Z_next=exp(Delta*Delta_t)Z + B_in*u_t, with Delta constrained to have negative symmetric part, mirroring the paper's auxiliary matrix Delta and the stable OU realization. The mathematical quantities are the positive decay constraint and Prony recurrence; empirical quantities are R_hat, the fitted rates, and the low-rank amplitude factors. Start with a 125M-parameter language model on WikiText-103 or a 1D Long Range Arena task, comparing against full causal attention and a same-width diagonal SSM at matched training FLOPs. Measure validation perplexity or accuracy, throughput, peak activation memory, and performance as sequence length grows. Success means near-attention accuracy while reducing temporal-state memory from O(Td) to O(Kr) and improving throughput at long context; also verify that hidden-state norms remain bounded when extrapolating to sequences 4x longer than training.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built and numerically tested a Prony memory bank with positive decay factors, bounded recurrent states, and nonlinear positive-amplitude Prony fitting. The three mechanism predictions were confirmed: half-life error was 2.18%, bounded-state ratios reached 1.0 across rho values, and fitting error collapsed from 2.65e-3 at K=1 to 1.53e-30 at K=3, matching the true mode count. On the synthetic long-memory task, the bank achieved much lower validation MSE than causal attention, but the unoptimized Python recurrence was slower, so an accuracy/mechanism win was observed but a throughput win was not demonstrated.

Agent confidence: 8/10

Baseline: Causal attention: T=32 validation MSE 4.1785, 2.136 s; T=128 validation MSE 6.4614, 4.525 s; 865 parameters.
Idea: Prony bank: T=32 validation MSE 2.241e-5, 7.687 s; T=128 validation MSE 0.00602, 29.802 s; 851 parameters. Math checks: half-life maximum relative error 2.18%, bounded-memory final/bound ratio 1.0, and K=3 Prony MSE 1.53e-30.

Limitations: Only a synthetic scalar long-memory regression task was tested; no language, speech, or Long Range Arena benchmark, no optimized fused recurrence, no peak-memory measurement, and no true matched-FLOP production comparison.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.05151) Sep 2, 2026

Significantly WORSE than the tuned baseline (+54.1%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.1347
Idea mean: 0.2076
Effect (Δ): 0.0729 (+54.1%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the registered sequence track was tested; throughput, peak memory, long-context scaling, and direct autocorrelation calibration were not tested.

How to run: PYTHONPATH=/home/maxwelhelp/all/math2nn /home/maxwelhelp/main/bin/python3 bench_prony.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 10/10

The Prony Memory Bank was evaluated on the registered sequence track with matched transformer_tiny structure, tuned baseline sweep, shared learning-rate search space, and 8 paired seeds. Baseline MSE was 0.1347019309 versus 0.2076336704 for the idea; delta_mean=+0.0729317395 and permutation p=0.0081, so the idea was significantly worse. Learned decay rates stayed below one, but the trained-model state-ratio signature was not confirmed.

Baseline: best lr=0.001; mean MSE=0.13470193091779947
Idea: K=8, lr=0.001; mean MSE=0.20763367041945457; delta=0.0729317395016551; p=0.0081

Benchmark result