Boundary-Hankel Mediator Regularization

Integrate this at training time around a small candidate mediator block, such as a 2-layer residual MLP inserted between transformer blocks or an MoE expert group. Let the candidate hidden state be z with dimension d and let the surrounding activations provide u. Use a local linearization of the candidate-plus-return path over a minibatch: estimate F by least squares from consecutive candidate states, $F=\arg\min_F\sum_t\|z_{t+1}-Fz_t\|_2^2$, estimate B from the regression residual dependence on external input u, and estimate C by regressing the returned external activation y on z. A simpler implementation can obtain F, B, and C directly from autograd Jacobian-vector products of the computational graph, averaged over a minibatch. For each candidate and every K training steps, run T=3 or 4 Gramian recurrences: initialize $R_0=0,O_0=0$ and update $R_{k+1}=FR_kF^\top+BB^\top$, $O_{k+1}=F^\top O_kF+C^\top C$. Construct the finite controllability and observability matrices, compute the small d-by-d product $H_T$ or its equivalent balanced singular values, and add $\alpha\mathcal{L}_{\mathrm{GMW}}$ to the task loss. Stop gradients through the estimated Jacobians initially to reduce instability, then allow them after the metric is validated. Estimate the mathematical quantities from minibatch Jacobians; do not assume anatomical or token adjacency. The first experiment should use a 6-layer small Transformer on CIFAR-10 or WikiText-2, with one candidate residual MLP and matched parameter/FLOP baselines. Compare no regularizer, reachability-only, observability-only, and the full Hankel term. Measure validation accuracy or perplexity, gradient norm variance, singular-value spectrum, and ablations that create one-sided receivers or broadcasters by zeroing B or C. Success means improved validation performance at equal compute, reduced seed variance, and a nontrivial increase in normalized Hankel energy and effective rank without exploding activation gain.