Fake-Stationary Volterra Memory Layer
Source paper: On (fake) Stationarity in Stochastic Volterra Equations with Affine Drift and Regular Kernels arXiv:2608.31099 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace a one-step recurrent update with a causal convolution over past affine innovations using an exponential-fractional kernel. Add mean reversion and calibrate the innovation amplitude so that activation mean and variance remain approximately invariant across sequence position while retaining long-range, power-law-like memory.
Formulas
Mathematical statement
The paper studies the stochastic Volterra equation $$X_t=X_0\phi(t)+\int_0^tK(t-s)(\mu(s)-\lambda X_s)\,ds+\int_0^tK(t-s)\sigma(s,X_s)\,dW_s,$$ where $X_t$ is the latent state, $X_0$ is an initial random variable independent of Brownian motion $W$, $\phi$ is a deterministic initial-condition response, $K$ is a causal memory kernel, $\mu(s)$ is the drift target, $\lambda>0$ is the mean-reversion strength, and $\sigma$ is the diffusion amplitude. The transferable structure is the separation of a long-memory linear response $K*(\mu-\lambda X)$ from a separately scaled innovation convolution. Use the paper's exponential-fractional kernel $$K(r)=\frac{e^{-\gamma r}r^{\alpha-1}}{\Gamma(\alpha)},\qquad r>0,\quad \alpha\geq1,\ \gamma\geq0,$$ where $\alpha$ controls regularity and memory shape and $\gamma$ controls exponential decay. For a neural layer, $u_t$ is the input feature, $\mu_t=g_\mu(u_t)$ is a learned drift target, $q_t=g_\sigma(u_t)>0$ is a base innovation scale, and $\sigma_t=\varsigma_tq_t$. With timestep $\Delta$, Gaussian innovations $\epsilon_t\sim\mathcal N(0,I)$, and discretized kernel weights $K_{t-s}=K((t-s)\Delta)$, use $$h_t=\phi_t h_0+\Delta\sum_{s=0}^{t-1}K_{t-s}(\mu_s-\lambda h_s)+\sqrt{\Delta}\sum_{s=0}^{t-1}K_{t-s}\sigma_s\epsilon_s.$$ Estimate the current variance $\widehat v_t$ over batch and feature dimensions and set $$\varsigma_t=\operatorname{clip}\left(\sqrt{\frac{v_\star}{\widehat v_t+\varepsilon}},\varsigma_{\min},\varsigma_{\max}\right),$$ where $v_\star$ is a target variance and $\varepsilon>0$ avoids division by zero. This is a neural approximation of the paper's deterministic stabilizing-factor construction.
Implementation notes
1. Integration point: implement the layer between an input projection and an output projection in a small language model, time-series forecaster, or diffusion latent model. Given input features $u_t\in\mathbb R^d$, compute $\mu_t=W_\mu u_t+b_\mu$ and $q_t=\operatorname{softplus}(W_\sigma u_t+b_\sigma)$. Maintain hidden states $h_t\in\mathbb R^d$ and return $y_t=W_o\operatorname{LayerNorm}(h_t)+u_t$.
2. Pseudocode: precompute $k_l=K(l\Delta)$ for $l=1,\ldots,T$ from the exponential-fractional formula. Sample $\epsilon_t\sim\mathcal N(0,I)$ during training and set $\epsilon_t=0$ for deterministic evaluation. Initialize $h_0\sim\mathcal N(0,v_\star I)$. For each position $t$, calculate $a_t=\Delta\sum_{s<t}k_{t-s}(\mu_s-\lambda h_s)$ and $b_t=\sqrt{\Delta}\sum_{s<t}k_{t-s}(\varsigma_sq_s\epsilon_s)$, then set $h_t=\phi_th_0+a_t+b_t$. Compute the convolutions with FFT for full sequences, or truncate to the previous $L$ positions for a bounded-memory implementation.
3. The paper supplies the mathematical components: causal Volterra convolution, affine mean reversion, and the exponential-fractional kernel. Estimate $\lambda$, $\alpha$, $\gamma$, $v_\star$, and clipping bounds by validation. Initialize $\lambda$ so $\lambda\Delta$ lies between $0.01$ and $0.2$. Use an exponential moving average of $\widehat v_t$ rather than raw batch variance, and stop gradients through the variance factor initially to avoid a second-order feedback loop. Log activation mean, variance, gradient norm, and empirical autocorrelation by sequence position.
4. First cheap experiment: train a 2-layer, 128-dimensional sequence model on Copying Memory and permuted-MNIST at lengths 256, 1024, and 4096. Compare a GRU, a diagonal state-space layer, the Volterra layer without variance calibration, and the calibrated layer at matched hidden size and optimizer. Measure validation accuracy, loss, activation-variance drift between the first and final sequence quarters, gradient explosions, peak memory, and wall-clock time. A positive result is at least 10% lower long-context error, substantially slower autocorrelation decay, and stable activation variance without more than 30% training overhead.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.