# Invariant-Preserving Latent Compression

- ID: 1635
- Canonical URL: https://synthcore.org/idea/1635/invariant-preserving-latent-compression
- API JSON: https://synthcore.org/api/idea/1635.json
- API Markdown: https://synthcore.org/api/idea/1635.md
- Verification status: unverified
- Source: [arXiv:2608.00397](https://arxiv.org/abs/2608.00397)
- Category: architecture
- Solves: memory, stability, accuracy
- ML areas: ssm, world-model, memory, diffusion
- Math tags: dynamical-systems, stochastic-processes, linear-algebra, numerical-analysis
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Replace unconstrained low-rank compression of a neural state with an augmented basis that always contains vectors representing known conserved quantities or diagnostically important linear statistics. After each learned transition, project the state back onto the affine constraint set with an exact minimum-norm correction, preventing rank truncation and model error from accumulating in those statistics.

## Mathematical statement

The paper starts from the stochastic transport equation $df+(v\cdot\nabla_x f+E\cdot\nabla_v f)dt+\sum_{k=1}^{K}\sigma_k(x)\cdot\nabla_v f\star d\beta_k=0$, where $f(t,x,v)$ is the distribution, $E$ is the self-consistent field, $\sigma_k$ are transport-noise coefficients, and $\beta_k$ are independent Brownian motions. Its stated construction fixes velocity modes associated with mass, momentum, and energy, augments the evolving low-rank basis with those modes, and performs conservative rank truncation. For a neural state $z\in\mathbb{R}^N$, represent the selected invariants by columns of $B\in\mathbb{R}^{N\times m}$ and their target values by $c\in\mathbb{R}^m$, so the constraints are $B^Tz=c$. Given a predicted state $y$, the closest exact correction is $z=y+B(B^TB)^{-1}(c-B^Ty)$, assuming $B$ has full column rank. For low-rank storage, let $U_r$ be the retained basis and let $Q=\operatorname{orth}([U_r,B])$; compression uses $Q$ rather than $U_r$, so invariant directions cannot be removed. If the constraints are not exactly constant, replace $c$ by a separately predicted or measured target $c_{n+1}$.

## Key formulas

- $$df(t,x,v)+\left(v\cdot\nabla_{x}f(t,x,v)+E(t,x)\cdot\nabla_{v}f(t,x,v)\right)dt+\sum_{k=1}^{K}\sigma_{k}(x)\cdot\nabla_{v}f(t,x,v)\star d\beta_k(t)=0.$$
- $$Q=\operatorname{orth}\!\left([U_r\;B]\right),\qquad z_{\mathrm{compressed}}=QQ^T y,$$
- $$z_{n+1}=y_{n+1}+B(B^TB)^{-1}\left(c_{n+1}-B^Ty_{n+1}\right),\qquad B^Tz_{n+1}=c_{n+1}.$$
- $$\min_{z\in\mathbb{R}^{N}}\frac12\|z-y_{n+1}\|_2^2\quad\mathrm{s.t.}\quad B^Tz=c_{n+1}.$$

## Implementation notes

(1) Integration point: use this in a latent SSM, neural operator, or diffusion/world-model transition whose hidden physical state is compressed to rank $r$. Let the flattened state be $z_n\in\mathbb{R}^N$, let the transition produce $y_{n+1}=F_\theta(z_n,u_n,\xi_n)$, and let $B$ contain known invariant directions. For a kinetic grid, columns can be discrete mass, each momentum component, and energy weights; for a generic system, use conserved charges, total mass, probability, or prescribed linear diagnostics. (2) Pseudocode: precompute full-rank $B$ and $G=(B^TB)^{-1}$. At each step compute `y = transition_theta(z, input, noise)`. If using a rank-$r$ state representation, maintain $U_r$, form `Q = qr(concat(U_r, B)).Q`, and replace the state or update subspace by `Q Q.T y`; when a strict target is required, set `z = y + B @ G @ (c_next - B.T @ y)`. Recompress only after augmentation: compute an SVD or randomized SVD in the orthogonal complement of $B$, retain the top $r-m$ residual directions, then concatenate them with $B$ and orthonormalize. (3) Compute $B$ and the correction matrix analytically from the chosen statistics. Estimate only the residual basis and singular values empirically; use QR plus randomized SVD for large $N$. If invariants are unknown, first fit candidate linear statistics on simulated trajectories and retain only those with low empirical variance. (4) First experiment: train a rank-16 latent recurrent neural operator on a 2-D advection or particle-grid dataset with a known conserved total mass and momentum, comparing ordinary rank-16 SVD compression against augmented rank-16 compression with $m=3$ invariant columns. Measure long-horizon rollout error, invariant drift, memory, and wall-clock cost. Success means invariant drift is reduced by at least 10x and long-horizon error improves at equal rank, with less than 15% additional transition cost; a stronger result is matching the unconstrained model's short-horizon error while using a smaller rank.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.