# Gramian-balanced neural SSM compression

- ID: 57
- Canonical URL: https://synthcore.org/idea/57/gramian-balanced-neural-ssm-compression
- API JSON: https://synthcore.org/api/idea/57.json
- API Markdown: https://synthcore.org/api/idea/57.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.22406](https://arxiv.org/abs/2608.22406)
- Category: memory
- Solves: memory, inference-speedup, accuracy
- ML areas: ssm, memory, inference-speedup, fine-tuning
- Math tags: linear-algebra, dynamical-systems, control-theory, numerical-analysis
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Compress the hidden state of a stable neural state-space layer using low-rank controllability and observability Gramians. States that are difficult to excite from the input or weakly visible at the output are removed, producing a smaller recurrent state with a principled input-output preservation criterion.

## Mathematical statement

For a discrete stable SSM with state update $x_{t+1}=Fx_t+Bu_t$ and output $y_t=Cx_t$, the paper defines controllability and observability Gramians through the Stein equations $FPF^{\top}-P+BB^{\top}=0$ and $F^{\top}QF-Q+C^{\top}C=0$; these are equations (1) and (2) specialized from the paper's generalized form by setting $E=I$ and $F=A$. Stability means every eigenvalue of $F$ lies strictly inside the unit circle, ensuring unique positive-semidefinite solutions. Compute low-rank factors $P\approx Z_pZ_p^{\top}$ and $Q\approx Z_qZ_q^{\top}$. Form $M=Z_q^{\top}Z_p$ and its SVD $M=U\Sigma V^{\top}$. For retained rank $r$, define $V_r=Z_pV_{:,1:r}\Sigma_{1:r,1:r}^{-1/2}$ and $W_r=Z_qU_{:,1:r}\Sigma_{1:r,1:r}^{-1/2}$, giving approximately $W_r^{\top}V_r=I$. The reduced matrices are $F_r=W_r^{\top}FV_r$, $B_r=W_r^{\top}B$, and $C_r=CV_r$. The preserved transfer function is $G(z)=C(zI-F)^{-1}B$, a specialization of the paper's $G(z)=C(zE-A)^{-1}B$. Low-rank factors can be generated without dense Gramians using $P_k=\sum_{i=0}^{k-1}F^iBB^{\top}(F^{\top})^i$ and the block factor iteration $Z_{p,k+1}=\operatorname{compress}_{\tau}([FZ_{p,k}\;B])$; apply the analogous iteration with $F^{\top}$ and $C^{\top}$ for $Q$. Here $\tau$ is a singular-value truncation tolerance and $\operatorname{compress}_{\tau}$ returns a thin factor after truncated or randomized SVD.

## Key formulas

- $$G(z)=C(zE-A)^{-1}B,\qquad z=e^{j\omega},$$
- $$APA^{\top}-EPE^{\top}+BB^{\top}=0,\qquad A^{\top}QA-E^{\top}QE+C^{\top}C=0,$$
- $$P=FPF^{\top}+BB^{\top},\qquad Q=F^{\top}QF+C^{\top}C\quad (E=I,\;F=A),$$
- $$M=Z_q^{\top}Z_p=U\Sigma V^{\top},\quad V_r=Z_pV_{:,1:r}\Sigma_r^{-1/2},\quad W_r=Z_qU_{:,1:r}\Sigma_r^{-1/2},\quad F_r=W_r^{\top}FV_r.$$

## Implementation notes

(1) Integrate this as an offline compression pass for a linear or nearly linear SSM layer. Expose its hidden-state transition $F\in\mathbb{R}^{n\times n}$, input matrix $B\in\mathbb{R}^{n\times m}$, and output matrix $C\in\mathbb{R}^{p\times n}$. If the layer has nonlinear gates, freeze the gates on representative calibration sequences or apply the method to the underlying linear transition. Require spectral radius $\rho(F)<1$; estimate it with 10 to 20 power iterations and scale $F$ by $\min(1-\epsilon,0.99/\max(\rho(F),10^{-6}))$ when necessary. (2) Pseudocode: initialize empty $Z_p$; for $k=1,\ldots,K$, set $Y=[FZ_p\;B]$ and $Z_p=\operatorname{compress}_{\tau}(Y)$; independently initialize empty $Z_q$ and repeat with $Y=[F^{\top}Z_q\;C^{\top}]$. Compute $M=Z_q^{\top}Z_p$, run its SVD, retain the first $r$ singular values, construct $V_r,W_r$, and replace $(F,B,C)$ with $(F_r,W_r^{\top}B,CV_r)$. (3) The Stein equations and balancing projection are taken from the paper; the engineer estimates factors matrix-free through products with $F$ and $F^{\top}$, never forming $P$ or $Q$. Monitor approximate residuals using $P=Z_pZ_p^{\top}$ and $Q=Z_qZ_q^{\top}$, and choose $K$ until residual norms stop decreasing. (4) First experiment: compress a 256- or 512-state S4- or Mamba-style layer to ranks 32, 64, and 128 on WikiText-103 or Long Range Arena, comparing against uncompressed fine-tuning and magnitude-based state pruning. Measure validation perplexity or task accuracy, activation memory, recurrent FLOPs, and decoding latency. A useful success criterion is at least 2x lower state-update cost and memory with less than 1% relative perplexity degradation after brief fine-tuning; also evaluate $\|G(e^{j\omega})-G_r(e^{j\omega})\|$ on a held-out frequency grid.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a NumPy/SciPy Gramian-balanced SSM compression MVP with matrix-free factor iteration, balancing projections, Stein residual checks, frequency-response evaluation, and magnitude-pruning comparison. The mechanism manifested: finite-horizon Gramian tails matched the exact predicted formula to numerical precision, scalar Gramian scaling matched 1/(1-rho^2) across rho=0.5–0.995, and every measured balanced-truncation frequency error stayed below the 2-times-discarded-Hankel-singular-values bound. At rank 4, Gramian compression had maximum frequency error 0.0988 versus 13.02 for magnitude pruning, with an ideal transition-cost ratio of 0.111.

### Mechanism check

- Verdict: Built a NumPy/SciPy Gramian-balanced SSM compression MVP with matrix-free factor iteration, balancing projections, Stein residual checks, frequency-response evaluation, and magnitude-pruning comparison. The mechanism manifested: finite-horizon Gramian tails matched the exact predicted formula to numerical precision, scalar Gramian scaling matched 1/(1-rho^2) across rho=0.5–0.995, and every measured balanced-truncation frequency error stayed below the 2-times-discarded-Hankel-singular-values bound. At rank 4, Gramian compression had maximum frequency error 0.0988 versus 13.02 for magnitude pruning, with an ideal transition-cost ratio of 0.111.
- Confidence: 9/10
- Limitations: Only small linear NumPy/SciPy toy systems were tested; no S4/Mamba layer, neural gates, WikiText/LRA task, fine-tuning, wall-clock latency, GPU execution, or measured activation-memory benchmark was evaluated. The cost ratio is an idealized state-transition ratio and excludes projection/implementation overhead.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.20345
- Paired wins: 2/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/697/bench_experiment.py)
- [bench_results/bench_report.json](https://synthcore.org/code/697/bench_results/bench_report.json)
- [experiment.py](https://synthcore.org/code/697/experiment.py)
- [report.md](https://synthcore.org/code/697/report.md)
- [report_bench_2026-08-31T222223.md](https://synthcore.org/code/697/report_bench_2026-08-31T222223.md)
- [results.json](https://synthcore.org/code/697/results.json)
- [Download all files as ZIP](https://synthcore.org/download/697)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.