# Folded-Gaussian Soft-Mode State Space

- ID: 2407
- Canonical URL: https://synthcore.org/idea/2407/folded-gaussian-soft-mode-state-space
- API JSON: https://synthcore.org/api/idea/2407.json
- API Markdown: https://synthcore.org/api/idea/2407.md
- Verification status: unverified
- Source: [arXiv:2608.16617](https://arxiv.org/abs/2608.16617)
- Category: memory
- Solves: memory, stability, accuracy
- ML areas: ssm, rnn, initialization, memory
- Math tags: statistical-mechanics, probability, spectral-theory, dynamical-systems
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Initialize a stable diagonal state-space layer with decay rates \(\omega_i=|\xi_i|\), where \(\xi_i\sim\mathcal N(\mu,\sigma^2)\), instead of using a narrowly clustered rate distribution. The nonzero density of rates near zero creates a population of slow modes whose aggregate impulse response has an algebraic tail, enabling long-horizon memory while every finite-dimensional mode remains exponentially stable.

## Mathematical statement

For \(\xi\sim\mathcal N(\mu,\sigma^2)\) and \(\omega=|\xi|\ge0\), the folded-Gaussian density is \(p(\omega)=[\exp(-(\omega-\mu)^2/(2\sigma^2))+\exp(-(\omega+\mu)^2/(2\sigma^2))]/(\sqrt{2\pi}\sigma)\) for \(\omega\ge0\). Its zero-frequency density is \(p(0)=2\exp[-\mu^2/(2\sigma^2)]/(\sqrt{2\pi}\sigma)\). For the diagonal continuous-time state model \(\dot h_i=-\omega_i h_i+b_i u\), every mode is asymptotically stable when \(\omega_i>0\). The population impulse response is \(K(t)=\mathbb E[e^{-\omega t}]=\int_0^\infty p(\omega)e^{-\omega t}d\omega\). Laplace asymptotics imply \(K(t)\sim p(0)/t\) when \(p(0)>0\). With timestep \(\Delta t\), the implementation uses \(a_i=e^{-\omega_i\Delta t}\in(0,1)\), so \(h_{k+1,i}=a_i h_{k,i}+b_i u_k\).

## Key formulas

- $$p(\omega)=\frac{1}{\sqrt{2\pi}\sigma}\left[\exp\left(-\frac{(\omega-\mu)^2}{2\sigma^2}\right)+\exp\left(-\frac{(\omega+\mu)^2}{2\sigma^2}\right)\right],\qquad \omega\ge0,$$
- $$p(0)=\frac{2}{\sqrt{2\pi}\sigma}\exp\left(-\frac{\mu^2}{2\sigma^2}\right),$$
- $$K(t)=\int_0^\infty p(\omega)e^{-\omega t}\,d\omega\sim\frac{p(0)}{t}\qquad (t\to\infty),$$
- $$h_{k+1,i}=e^{-\omega_i\Delta t}h_{k,i}+b_i u_k,\qquad \omega_i=|\xi_i|,\quad \xi_i\sim\mathcal N(\mu,\sigma^2).$$

## Implementation notes

1. Integration point: implement this as a replaceable diagonal SSM or gated RNN memory block before a task-specific MLP. Use \(N\) channels, state \(h_k\in\mathbb R^N\), input projection \(b\), output projection \(c\), and output \(y_k=c^\top h_k\). Keep the rates fixed initially so the experiment isolates the spectral mechanism. 2. Pseudocode: sample \(\xi_i\sim\mathcal N(\mu,\sigma^2)\), set \(\omega_i=|\xi_i|+\epsilon\), set \(a_i=\exp(-\omega_i\Delta t)\), and at each sequence step compute \(h_i\leftarrow a_i h_i+b_i u_k\), followed by \(y_k=c^\top h_k\). Optionally learn \(\mu\) and \(\sigma\) using softplus parameters, while regularizing the empirical low-rate density toward the analytic \(p(0)\). 3. Computed from the paper: the folded density, pole stability condition \(0<a_i<1\), and predicted tail prefactor \(p(0)\). Estimated empirically: the impulse response \(K_k=N^{-1}\sum_i a_i^k\), its local log-log slope, and the minimum sampled rate. 4. First cheap experiment: train a one-layer memory block plus linear readout on delayed-copy and adding tasks with sequence length 1,000. Compare folded-Gaussian rates with uniform-log rates and a GRU at equal parameter count. Before training, inject an impulse and measure \(K_k\). The predicted signature is a log-log slope near \(-1\), with fitted prefactor agreeing with \(2e^{-\mu^2/(2\sigma^2)}/(\sqrt{2\pi}\sigma)\) within 20% over the intermediate asymptotic regime. Increasing \(\mu/\sigma\) should exponentially suppress the tail and produce a measurable transition toward short-memory behavior. Check that all discrete poles remain below one.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.