Balanced State-Order Compression

Integrate this as an offline compression pass for every wide hidden layer of an MLP or Transformer feed-forward block; do not compute the Gramians during ordinary training. First train a baseline model and collect $N$ representative deployment-distribution samples. Run the model while saving the activation tensor $h_\ell^{(i)}$ at each candidate layer, compute its mean $\bar h_\ell$, and estimate $W_c$ from centered outer products. Estimate observability by computing Jacobian-vector products of the output logits with respect to $h_\ell$. For small models, materialize $J_i$ directly. For larger models, sample Rademacher vectors $q_k\in\mathbb{R}^{d_z}$ and accumulate $(J_i^\top q_k)(J_i^\top q_k)^\top$; its expectation is $J_i^\top J_i$. Add $\epsilon I$ before taking matrix square roots. Compute leading eigenvectors of $M=W_c^{1/2}W_oW_c^{1/2}$ with randomized Lanczos, choose $r$ using $\rho\in\{0.99,0.995,0.999\}$, and set $V_r=W_c^{1/2}U_{:,1:r}$ followed by QR. Replace the state by $\tilde h_\ell=V_r^\top(h_\ell-\bar h_\ell)$. If the following linear map is $Wh+b$, replace it with $WV_r\tilde h+(b+W\bar h)$. For nonlinear blocks, insert the projection after the activation and adjust adjacent layers, then fine-tune for 1--5 epochs using task loss plus a baseline-logit distillation loss. The mathematics supplies the Gramians, balanced matrix, eigenvalue ranking, and projection; sample selection, Jacobian approximation, $\epsilon$, $\rho$, and fine-tuning are empirical. Start with a four-layer SiLU MNIST MLP and compare balanced truncation against PCA/SVD and unstructured pruning at equal parameter count. Measure accuracy, parameter count, activation memory, and GPU latency. A successful result is 50--70% parameter reduction with less than 0.5 percentage-point accuracy loss and lower inference latency.