# Gramian-Regularized Latent State Models

- ID: 1350
- Canonical URL: https://synthcore.org/idea/1350/gramian-regularized-latent-state-models
- API JSON: https://synthcore.org/api/idea/1350.json
- API Markdown: https://synthcore.org/api/idea/1350.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.24134](https://arxiv.org/abs/2607.24134)
- Category: regularization
- Solves: stability, accuracy, sample-efficiency
- ML areas: ssm, rnn, world-model, training-dynamics
- Math tags: control-theory, dynamical-systems, linear-algebra, optimization
- Ratings: usefulness 7/10; difficulty 6/10; novelty 6/10

## Idea description

Add finite-horizon observability and reachability objectives to a recurrent or state-space neural model so that its latent modes are both inferable from outputs and influenceable by available inputs. This directly penalizes the failure mode identified in the paper: a large latent perturbation with nearly zero first-order output projection.

## Mathematical statement

Let the learned latent dynamics be $\dot z=A(t)z+B(t)u$ and measurements be $y=C(t)z$, where $z\in\mathbb{R}^{n}$ is the latent state, $u\in\mathbb{R}^{m}$ is the input, $y\in\mathbb{R}^{p}$ is the output, and $\Phi(t,\tau)$ is the state-transition matrix satisfying $\partial_t\Phi(t,\tau)=A(t)\Phi(t,\tau)$ and $\Phi(\tau,\tau)=I$. The finite-time observability Gramian is $W_o=\int_{t_0}^{t_1}\Phi(t,t_0)^TC(t)^TC(t)\Phi(t,t_0)\,dt$, and the reachability Gramian is $W_r=\int_{t_0}^{t_1}\Phi(t_1,\tau)B(\tau)B(\tau)^T\Phi(t_1,\tau)^T\,d\tau$. The smallest eigenvalues quantify hidden and unreachable latent directions. Use penalties $\max(0,\epsilon-\lambda_{\min}(W_o))$ and $\max(0,\epsilon-\lambda_{\min}(W_r))$, with trace normalization to control scale.

## Key formulas

- $$W_o(t_0,t_1)=\int_{t_0}^{t_1}\Phi(t,t_0)^T C(t)^T C(t)\Phi(t,t_0)\,dt,$$
- $$W_r(t_0,t_1)=\int_{t_0}^{t_1}\Phi(t_1,\tau)B(\tau)B(\tau)^T\Phi(t_1,\tau)^T\,d\tau,$$
- $$\mathcal{L}=\mathcal{L}_{task}+\lambda_o\max\!\left(0,\epsilon-\frac{\lambda_{\min}(W_o)}{\operatorname{tr}(W_o)/n}\right)+\lambda_r\max\!\left(0,\epsilon-\frac{\lambda_{\min}(W_r)}{\operatorname{tr}(W_r)/n}\right).$$

## Implementation notes

Integrate this into a GRU, continuous-time neural ODE, or state-space model with latent state $z_t$. For each training window, linearize the transition and readout around the current rollout: compute $A_t=\partial f_\theta/\partial z$, $B_t=\partial f_\theta/\partial u$, and $C_t=\partial h_\theta/\partial z$. Initialize $\Phi=I$, $W_o=0$, and $W_r=0$. For each timestep, update $\Phi_{t+1}=(I+\Delta t A_t)\Phi_t$ and accumulate $W_o\mathrel{+}=\Delta t\,\Phi_t^TC_t^TC_t\Phi_t$. Use backward transition propagation $\Psi_t=\Psi_{t+1}(I+\Delta t A_t)$ to accumulate $W_r\mathrel{+}=\Delta t\,\Psi_{t+1}B_tB_t^T\Psi_{t+1}^T$. Estimate minimum eigenvalues with 5--10 Lanczos or power iterations instead of a full eigendecomposition. Add the normalized penalties to the task loss and clip gradients from the Gramian term. The paper supplies the finite-time Gramian mechanism; neural Jacobians and finite-window approximations are estimated from minibatch rollouts. First experiment: compare a 32-dimensional GRU with and without the penalties on permuted-MNIST and a synthetic delayed-copy task. Sweep $\lambda_o$ and $\lambda_r$. The predicted signature is a sharp performance degradation when normalized $\lambda_{\min}(W_o)$ falls below approximately $\epsilon=0.05$; regularization should prevent this collapse. Near the weak-observability boundary, latent reconstruction error should scale approximately as $1/\lambda_{\min}(W_o)$.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built and numerically verified finite-horizon observability/reachability Gramian penalties in a differentiable 2D linear state-space model. The predicted normalized minimum-eigenvalue scaling matched exactly, continuous-time quadrature agreed with analytic integrals, and reconstruction MSE scaled as lambda_min^-0.954, close to the predicted inverse law. In the tiny training comparison, regularization improved normalized reachability but did not improve observability or task loss, so the mechanism is confirmed while a practical accuracy win was not observed.

### Mechanism check

- Verdict: Built and numerically verified finite-horizon observability/reachability Gramian penalties in a differentiable 2D linear state-space model. The predicted normalized minimum-eigenvalue scaling matched exactly, continuous-time quadrature agreed with analytic integrals, and reconstruction MSE scaled as lambda_min^-0.954, close to the predicted inverse law. In the tiny training comparison, regularization improved normalized reachability but did not improve observability or task loss, so the mechanism is confirmed while a practical accuracy win was not observed.
- Confidence: 8/10
- Limitations: Only a 2D linear SSM and synthetic finite-window task were tested; no GRU, neural ODE, permuted-MNIST, hyperparameter sweep, minibatch Jacobian linearization, Lanczos approximation, or larger-scale accuracy/generalization evaluation was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.5437
- Paired wins: 4/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/529/bench_report.json)
- [gramian_experiment.py](https://synthcore.org/code/529/gramian_experiment.py)
- [report.md](https://synthcore.org/code/529/report.md)
- [report_bench_2026-08-31T171946.md](https://synthcore.org/code/529/report_bench_2026-08-31T171946.md)
- [results.json](https://synthcore.org/code/529/results.json)
- [stage2_gramian_bench.py](https://synthcore.org/code/529/stage2_gramian_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/529)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.