LP-Synthesized Bounded Residual State

(1) Integration point: implement this as a recurrent residual block between transformer or MLP layers, or as a small diagonal state-space module. Let $x_t\in\mathbb{R}^m$ be a normalized feature disturbance or current layer embedding, let $z_t\in\mathbb{R}^N$ be the bounded latent state, and let $y_t=Hz_t+r_t$ be the feature correction added to the main network. Use a fixed sparse mask $\mathcal{M}$ to determine which state coordinates may communicate. Estimate $b_j$ from a calibration set using a high quantile plus a safety factor, and estimate $\ell_i$ as the maximum absolute residual after fitting $H$. (2) Solve an offline LP. Variables are $U$, $r_z$, and auxiliary nonnegative variables $S_{ij}\geq|U_{ij}|$. Minimize $\sum_i r_{z,i}+\lambda\sum_{ij}S_{ij}$ subject to $-S\leq U\leq S$, $U_{ij}=0$ outside $\mathcal{M}$, $r_{z,i}\geq|z_{0,i}|$, $r_{z,i}\geq\sum_jS_{ij}b_j/\alpha_i$, optional gain bounds $-V\leq U\leq V$, and output constraints $\sum_j|H_{ij}|r_{z,j}+\ell_i\leq\mu_i$. Convert the continuous update to code with Euler step $z_{t+1}=(1-\Delta\alpha)\odot z_t+\Delta Ux_t$, requiring $0<\Delta\alpha_i\leq1$; alternatively use exact diagonal decay $z_{t+1}=e^{-\Delta\alpha}\odot z_t+(1-e^{-\Delta\alpha})\odot(Ux_t/\alpha)$. (3) The LP supplies $U$ and certified radii; empirical calibration supplies $b$, $\ell$, $H$, and the step size. During training, optionally penalize certificate slack $\sum_i\operatorname{ReLU}(|y_i|-\mu_i)$, but do not rely on it for the guarantee. (4) First experiment: compare a 2-layer GRU or diagonal SSM with this module on sequential CIFAR-10, Speech Commands, or a small character language-model task under Gaussian, adversarial, and held-out distribution-shift perturbations. Match parameter count and FLOPs against a standard recurrent residual block. Measure accuracy, hidden-state violation rate, gradient-norm variance, and robustness when test perturbations exceed calibration bounds by 10--50%. The expected signal is near-zero certified violations inside the bound, lower activation and gradient explosions, and equal or better accuracy under perturbation at similar compute.