# Prony Memory Bank for Linear-Time Sequence Modeling - ID: 538 - Canonical URL: https://synthcore.org/idea/538/prony-memory-bank-for-linear-time-sequence-modeling - API JSON: https://synthcore.org/api/idea/538.json - API Markdown: https://synthcore.org/api/idea/538.md - Verification status: failed_benchmark - Source: [arXiv:2607.05151](https://arxiv.org/abs/2607.05151) - Category: architecture - Solves: speedup, memory, stability, accuracy - ML areas: attention, ssm, inference-speedup, memory - Math tags: stochastic-processes, dynamical-systems, linear-algebra, spectral-theory - Ratings: usefulness 7/10; difficulty 5/10; novelty 4/10 ## Idea description Replace quadratic self-attention over a sequence with a bank of K auxiliary exponentially decaying states whose rates are fitted directly from the empirical autocorrelation of the sequence features. Each mode captures a distinct time scale, so the module can represent short- and long-range dependencies with O(TK) computation and O(K) recurrent memory rather than storing all previous tokens. Constrain decay rates to be positive and use the paper's extended Markovian block structure to obtain a stable recurrent realization. ## Mathematical statement Equation (22)/(35) constructs an extended Markov process with velocity-like variable Y in R^d, auxiliary state Z in R^(N-2d), and position-like variable X in R^d: d[Y;Z;X] = [[0,B^T,-Lambda_0/m],[-C,Delta,0],[I,0,0]][Y;Z;X]dt + [0;L;0]dW. Here B and C couple the observable and auxiliary states, Delta controls auxiliary-state decay, Lambda_0 is the effective stiffness matrix, m is a positive scalar mass, L is the noise factor, and W is Brownian motion. Proposition 7 states that Lambda_0 is symmetric positive definite, L_2=0, and the stationary covariance has the block form Sigma_0=(beta m)^(-1) diag(I,Sigma_22,m Lambda_0^(-1)); this motivates enforcing positive covariance and stable decay in the neural realization. The paper's data-driven principle is to approximate an observed autocorrelation matrix R(tau) by a finite Prony series R(tau) approximately equal to sum_{k=1}^K A_k exp(-lambda_k tau), where lambda_k>0 are decay rates and A_k are matrix amplitudes. For neural inputs u_t, use the adapted discrete Markov realization s_{k,t+1}=rho_k s_{k,t}+G_k u_t with rho_k=exp(-lambda_k Delta t), and output y_t=sum_k H_k s_{k,t}; positive lambda_k gives |rho_k|<1 and therefore bounded memory. ## Key formulas - $$\mathrm{d}\begin{bmatrix}Y\\ Z\\ X\end{bmatrix}=\begin{bmatrix}0&B^{T}&-\Lambda_{0}/m\\-C&\Delta&0\\I&0&0\end{bmatrix}\begin{bmatrix}Y\\ Z\\ X\end{bmatrix}\,\mathrm{d}t+\begin{bmatrix}0\\L\\0\end{bmatrix}\,\mathrm{d}W.$$ - $$\Sigma_{0}=\frac{1}{\beta m}\begin{bmatrix}I&0&0\\0&\Sigma_{22}&0\\0&0&m\Lambda_{0}^{-1}\end{bmatrix},\qquad \Lambda_{0}=mB_{1}^{T}C_{1},\qquad L_{2}=-C_{1}^{-1}L_{1}=0.$$ - $$R(\tau)\approx\sum_{k=1}^{K}A_{k}e^{-\lambda_{k}\tau},\qquad \lambda_k>0,$$ - $$s_{k,t+1}=\rho_k s_{k,t}+G_k u_t,\quad \rho_k=e^{-\lambda_k\Delta t},\quad y_t=\sum_{k=1}^{K}H_k s_{k,t}.$$ ## Implementation notes Integrate this as a causal replacement for a self-attention block in a transformer encoder, speech model, or time-series model. Let u_t in R^d be the normalized token embedding after a small input projection. Maintain K auxiliary states s_k in R^r, with K between 4 and 32 and r=d/K or a smaller bottleneck. First collect a calibration stream of hidden features from a baseline model and estimate the lag autocorrelation R_hat(j)=1/(T-j) sum_{t=1}^{T-j} u_t u_{t+j}^T for j=0,...,J. Fit the Prony model R_hat(j) approximately sum_k A_k rho_k^j by nonlinear least squares, parameterizing rho_k=sigmoid(a_k) so 0