# Balanced State-Order Compression

- ID: 437
- Canonical URL: https://synthcore.org/idea/437/balanced-state-order-compression
- API JSON: https://synthcore.org/api/idea/437.json
- API Markdown: https://synthcore.org/api/idea/437.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.05457](https://arxiv.org/abs/2607.05457)
- Category: architecture
- Solves: memory, speedup, accuracy
- ML areas: mlp, transformer, inference-speedup, pruning
- Math tags: control-theory, dynamical-systems, linear-algebra, spectral-theory
- Ratings: usefulness 8/10; difficulty 5/10; novelty 7/10

## Idea description

Compress each hidden layer by retaining directions that are simultaneously reachable from the observed input distribution and observable at the network output. Unlike PCA or SVD, the retained subspace is weighted by downstream task sensitivity, so high-variance but output-irrelevant directions can be removed while low-variance predictive directions are preserved.

## Mathematical statement

The network is treated as the depth-indexed nonlinear system $h_{\ell+1}=f_{\ell}(h_{\ell})$ and $z=g(h_L)$, where $h_{\ell}\in\mathbb{R}^{n_\ell}$ is the hidden state at layer $\ell$ and $z\in\mathbb{R}^{d_z}$ is the output. For centered hidden snapshots $x_i=h_\ell^{(i)}-\bar h_\ell$, the empirical reachability Gramian is $W_c=\frac{1}{N}\sum_{i=1}^{N}x_i x_i^\top$, measuring which state directions are excited by data. With $J_i=\partial z/\partial h_\ell\vert_{h_\ell^{(i)}}\in\mathbb{R}^{d_z\times n_\ell}$, the empirical observability Gramian is $W_o=\frac{1}{N}\sum_{i=1}^{N}J_i^\top J_i$, measuring which hidden perturbations change the output. Form $M=W_c^{1/2}W_oW_c^{1/2}=U\operatorname{diag}(\sigma_1,\ldots,\sigma_{n_\ell})U^\top$, where $\sigma_1\geq\cdots\geq\sigma_{n_\ell}\geq0$ are balanced energies. Select the smallest $r$ satisfying $\sum_{j=1}^{r}\sigma_j/\sum_{j=1}^{n_\ell}\sigma_j\geq\rho$, and retain $V_r=W_c^{1/2}U_{:,1:r}$ after QR orthonormalization.

## Key formulas

- $$h_{\ell+1}=f_{\ell}(h_{\ell}),\qquad z=g(h_{L})$$
- $$W_c^{(\ell)}=\frac{1}{N}\sum_{i=1}^{N}(h_\ell^{(i)}-\bar h_\ell)(h_\ell^{(i)}-\bar h_\ell)^\top$$
- $$W_o^{(\ell)}=\frac{1}{N}\sum_{i=1}^{N}J_i^\top J_i,\qquad J_i=\left.\frac{\partial z}{\partial h_\ell}\right|_{h_\ell^{(i)}}$$
- $$M_\ell=(W_c^{(\ell)})^{1/2}W_o^{(\ell)}(W_c^{(\ell)})^{1/2}=U\operatorname{diag}(\sigma_1,\ldots,\sigma_{n_\ell})U^\top,\qquad \frac{\sum_{j=1}^{r}\sigma_j}{\sum_{j=1}^{n_\ell}\sigma_j}\geq\rho$$

## Implementation notes

Integrate this as an offline compression pass for every wide hidden layer of an MLP or Transformer feed-forward block; do not compute the Gramians during ordinary training. First train a baseline model and collect $N$ representative deployment-distribution samples. Run the model while saving the activation tensor $h_\ell^{(i)}$ at each candidate layer, compute its mean $\bar h_\ell$, and estimate $W_c$ from centered outer products. Estimate observability by computing Jacobian-vector products of the output logits with respect to $h_\ell$. For small models, materialize $J_i$ directly. For larger models, sample Rademacher vectors $q_k\in\mathbb{R}^{d_z}$ and accumulate $(J_i^\top q_k)(J_i^\top q_k)^\top$; its expectation is $J_i^\top J_i$. Add $\epsilon I$ before taking matrix square roots. Compute leading eigenvectors of $M=W_c^{1/2}W_oW_c^{1/2}$ with randomized Lanczos, choose $r$ using $\rho\in\{0.99,0.995,0.999\}$, and set $V_r=W_c^{1/2}U_{:,1:r}$ followed by QR. Replace the state by $\tilde h_\ell=V_r^\top(h_\ell-\bar h_\ell)$. If the following linear map is $Wh+b$, replace it with $WV_r\tilde h+(b+W\bar h)$. For nonlinear blocks, insert the projection after the activation and adjust adjacent layers, then fine-tune for 1--5 epochs using task loss plus a baseline-logit distillation loss. The mathematics supplies the Gramians, balanced matrix, eigenvalue ranking, and projection; sample selection, Jacobian approximation, $\epsilon$, $\rho$, and fine-tuning are empirical. Start with a four-layer SiLU MNIST MLP and compare balanced truncation against PCA/SVD and unstructured pruning at equal parameter count. Measure accuracy, parameter count, activation memory, and GPU latency. A successful result is 50--70% parameter reduction with less than 0.5 percentage-point accuracy loss and lower inference latency.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built an offline balanced-Gramian compression pass with toy verification and a small digits MLP benchmark. The mechanism manifested exactly: balanced ordering followed the reachability-observability product rather than PCA variance, the predicted mode transition occurred at alpha=0.1, and observability scaling changed total balanced energy linearly while preserving rank. In the benchmark, balanced rank-32 retained baseline accuracy (0.9622 vs 0.9622), but it did not produce a practical win: estimated parameters fell about 11% and measured latency increased (0.632 ms vs 0.310 ms).

### Mechanism check

- Verdict: Built an offline balanced-Gramian compression pass with toy verification and a small digits MLP benchmark. The mechanism manifested exactly: balanced ordering followed the reachability-observability product rather than PCA variance, the predicted mode transition occurred at alpha=0.1, and observability scaling changed total balanced energy linearly while preserving rank. In the benchmark, balanced rank-32 retained baseline accuracy (0.9622 vs 0.9622), but it did not produce a practical win: estimated parameters fell about 11% and measured latency increased (0.632 ms vs 0.310 ms).
- Confidence: 9/10
- Limitations: Only one hidden layer, one fixed random seed, and the sklearn digits dataset were tested. No fine-tuning, transformer blocks, Jacobian-vector approximation, multi-layer end-to-end replacement, repeated statistical trials, or realistic deployment benchmark was evaluated; the reported compressed parameter count excludes any separately materialized projection cost.

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/92/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/92/bench_report.json)
- [experiment.py](https://synthcore.org/code/92/experiment.py)
- [report.md](https://synthcore.org/code/92/report.md)
- [report_bench_2026-08-30T121851.md](https://synthcore.org/code/92/report_bench_2026-08-30T121851.md)
- [results.json](https://synthcore.org/code/92/results.json)
- [Download all files as ZIP](https://synthcore.org/download/92)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.