{
 "artifacts": null,
 "category": "dynamics",
 "description": "Replace a one-step recurrent update with a causal convolution over past affine innovations using an exponential-fractional kernel. Add mean reversion and calibrate the innovation amplitude so that activation mean and variance remain approximately invariant across sequence position while retaining long-range, power-law-like memory.",
 "formulas_latex": [
  "$$X_t=X_0\\phi(t)+\\int_0^tK(t-s)(\\mu(s)-\\lambda X_s)\\,ds+\\int_0^tK(t-s)\\sigma(s,X_s)\\,dW_s.$$",
  "$$K(r)=\\frac{e^{-\\gamma r}r^{\\alpha-1}}{\\Gamma(\\alpha)},\\qquad r\u003e0,\\quad \\alpha\\geq1,\\ \\gamma\\geq0.$$",
  "$$h_t=\\phi_t h_0+\\Delta\\sum_{s=0}^{t-1}K_{t-s}(\\mu_s-\\lambda h_s)+\\sqrt{\\Delta}\\sum_{s=0}^{t-1}K_{t-s}\\sigma_s\\epsilon_s.$$",
  "$$\\sigma_t=\\varsigma_tq_t,\\qquad \\varsigma_t=\\operatorname{clip}\\left(\\sqrt{\\frac{v_\\star}{\\widehat v_t+\\varepsilon}},\\varsigma_{\\min},\\varsigma_{\\max}\\right),\\qquad \\widehat v_t=\\operatorname{Var}_{b,d}(h_{b,t,d}).$$"
 ],
 "id": 2917,
 "implementation": "1. Integration point: implement the layer between an input projection and an output projection in a small language model, time-series forecaster, or diffusion latent model. Given input features $u_t\\in\\mathbb R^d$, compute $\\mu_t=W_\\mu u_t+b_\\mu$ and $q_t=\\operatorname{softplus}(W_\\sigma u_t+b_\\sigma)$. Maintain hidden states $h_t\\in\\mathbb R^d$ and return $y_t=W_o\\operatorname{LayerNorm}(h_t)+u_t$.\n\n2. Pseudocode: precompute $k_l=K(l\\Delta)$ for $l=1,\\ldots,T$ from the exponential-fractional formula. Sample $\\epsilon_t\\sim\\mathcal N(0,I)$ during training and set $\\epsilon_t=0$ for deterministic evaluation. Initialize $h_0\\sim\\mathcal N(0,v_\\star I)$. For each position $t$, calculate $a_t=\\Delta\\sum_{s\u003ct}k_{t-s}(\\mu_s-\\lambda h_s)$ and $b_t=\\sqrt{\\Delta}\\sum_{s\u003ct}k_{t-s}(\\varsigma_sq_s\\epsilon_s)$, then set $h_t=\\phi_th_0+a_t+b_t$. Compute the convolutions with FFT for full sequences, or truncate to the previous $L$ positions for a bounded-memory implementation.\n\n3. The paper supplies the mathematical components: causal Volterra convolution, affine mean reversion, and the exponential-fractional kernel. Estimate $\\lambda$, $\\alpha$, $\\gamma$, $v_\\star$, and clipping bounds by validation. Initialize $\\lambda$ so $\\lambda\\Delta$ lies between $0.01$ and $0.2$. Use an exponential moving average of $\\widehat v_t$ rather than raw batch variance, and stop gradients through the variance factor initially to avoid a second-order feedback loop. Log activation mean, variance, gradient norm, and empirical autocorrelation by sequence position.\n\n4. First cheap experiment: train a 2-layer, 128-dimensional sequence model on Copying Memory and permuted-MNIST at lengths 256, 1024, and 4096. Compare a GRU, a diagonal state-space layer, the Volterra layer without variance calibration, and the calibrated layer at matched hidden size and optimizer. Measure validation accuracy, loss, activation-variance drift between the first and final sequence quarters, gradient explosions, peak memory, and wall-clock time. A positive result is at least 10% lower long-context error, substantially slower autocorrelation decay, and stable activation variance without more than 30% training overhead.",
 "math_summary": "The paper studies the stochastic Volterra equation $$X_t=X_0\\phi(t)+\\int_0^tK(t-s)(\\mu(s)-\\lambda X_s)\\,ds+\\int_0^tK(t-s)\\sigma(s,X_s)\\,dW_s,$$ where $X_t$ is the latent state, $X_0$ is an initial random variable independent of Brownian motion $W$, $\\phi$ is a deterministic initial-condition response, $K$ is a causal memory kernel, $\\mu(s)$ is the drift target, $\\lambda\u003e0$ is the mean-reversion strength, and $\\sigma$ is the diffusion amplitude. The transferable structure is the separation of a long-memory linear response $K*(\\mu-\\lambda X)$ from a separately scaled innovation convolution. Use the paper's exponential-fractional kernel $$K(r)=\\frac{e^{-\\gamma r}r^{\\alpha-1}}{\\Gamma(\\alpha)},\\qquad r\u003e0,\\quad \\alpha\\geq1,\\ \\gamma\\geq0,$$ where $\\alpha$ controls regularity and memory shape and $\\gamma$ controls exponential decay. For a neural layer, $u_t$ is the input feature, $\\mu_t=g_\\mu(u_t)$ is a learned drift target, $q_t=g_\\sigma(u_t)\u003e0$ is a base innovation scale, and $\\sigma_t=\\varsigma_tq_t$. With timestep $\\Delta$, Gaussian innovations $\\epsilon_t\\sim\\mathcal N(0,I)$, and discretized kernel weights $K_{t-s}=K((t-s)\\Delta)$, use $$h_t=\\phi_t h_0+\\Delta\\sum_{s=0}^{t-1}K_{t-s}(\\mu_s-\\lambda h_s)+\\sqrt{\\Delta}\\sum_{s=0}^{t-1}K_{t-s}\\sigma_s\\epsilon_s.$$ Estimate the current variance $\\widehat v_t$ over batch and feature dimensions and set $$\\varsigma_t=\\operatorname{clip}\\left(\\sqrt{\\frac{v_\\star}{\\widehat v_t+\\varepsilon}},\\varsigma_{\\min},\\varsigma_{\\max}\\right),$$ where $v_\\star$ is a target variance and $\\varepsilon\u003e0$ avoids division by zero. This is a neural approximation of the paper's deterministic stabilizing-factor construction.",
 "math_tags": [
  "stochastic-processes",
  "dynamical-systems",
  "probability",
  "numerical-analysis"
 ],
 "ml_areas": [
  "ssm",
  "rnn",
  "world-model",
  "diffusion"
 ],
 "paper": {
  "arxiv_id": "2608.31099",
  "arxiv_url": "https://arxiv.org/abs/2608.31099",
  "summary_what_math_gives_to_ml": "The paper gives a constructive recipe for combining causal Volterra memory, affine mean reversion, and a stabilizing diffusion amplitude so that marginal statistics remain approximately stationary despite long-range dependence. This can transfer to recurrent or state-space neural layers as a power-law memory block that preserves information over long contexts without the uncontrolled variance drift common in unconstrained recurrences. The most practical implementation is a discretized exponential-fractional convolution with learned drift targets and batch-estimated innovation scaling. The key experiment is whether this produces longer useful memory and more stable activations than GRUs or standard diagonal state-space models at comparable compute.",
  "title": "On (fake) Stationarity in Stochastic Volterra Equations with Affine Drift and Regular Kernels",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 6,
  "novelty": 5,
  "usefulness": 5
 },
 "solves": [
  "stability",
  "accuracy",
  "scalability"
 ],
 "title": "Fake-Stationary Volterra Memory Layer",
 "url": "https://synthcore.org/idea/2917/fake-stationary-volterra-memory-layer",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}