# LP-Synthesized Bounded Residual State

- ID: 2115
- Canonical URL: https://synthcore.org/idea/2115/lp-synthesized-bounded-residual-state
- API JSON: https://synthcore.org/api/idea/2115.json
- API Markdown: https://synthcore.org/api/idea/2115.md
- Verification status: unverified
- Source: [arXiv:2608.11141](https://arxiv.org/abs/2608.11141)
- Category: dynamics
- Solves: stability, generalization, accuracy
- ML areas: rnn, ssm, training, regularization
- Math tags: dynamical-systems, control-theory, optimization, linear-algebra
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace an unconstrained recurrent residual update with a sparse coordinated state-space block whose gains and state radii are synthesized jointly by a linear program. The block receives bounded feature disturbances, keeps every hidden coordinate inside a certified interval for all time, and uses an affine feedforward correction to reduce the output sensitivity of downstream coordinates.

## Mathematical statement

The paper's key certificate is the componentwise bound $|y_i(t)|\leq\tilde b_i$, with $\tilde b_i:=\sum_{j=1}^{N}|(HG)_{ij}|r_{z,j}+\tilde\ell_i$. Here $z\in\mathbb{R}^{N}$ is the virtual controller state, $r_{z,j}\geq0$ is its certified radius, $H$ is the feeder-to-output sensitivity matrix, $G$ is the coordination matrix, and $\tilde\ell_i$ bounds the local unmodeled disturbance at output $i$. The proof relies on the forward-invariant box $\mathcal{Z}=\{z\in\mathbb{R}^{N}:|z_i|\leq r_{z,i}\}$ and the relation $y=HGz+r$, where $|r_i|\leq\tilde\ell_i$. The design program defines $W:=HU$ and imposes sparse communication through $U_{ij}=0$ for $(i,j)\notin\mathcal{M}$, elementwise gain bounds $-V\leq U\leq V$, output bounds $-\mu+\bar\phi\leq y'_0\mathbf{1}+H(q_\phi+y'_0q_y)\leq\mu-\bar\phi$, and reserve constraints $-Q^{\max}+V\mathbf{1}\leq q_\phi+y'_0q_y\leq Q^{\max}-V\mathbf{1}$. It also requires $r_z\geq|z(0)|$, $r_z\geq Db$, and $r_z\geq\underline r$, where $D=\operatorname{diag}(d_1,\ldots,d_N)$ maps disturbance bounds $b$ into state excursions. The neural adaptation uses the same box-invariance logic: for $\dot z=-\operatorname{diag}(\alpha)z+Ux$, $|x_j|\leq b_j$, choose $r_z$ satisfying $r_{z,i}\geq|z_i(0)|$ and $\alpha_i r_{z,i}\geq\sum_j|U_{ij}|b_j$; then $|z_i(t)|\leq r_{z,i}$ for all $t\geq0$. The output certificate is $|y_i|\leq\sum_j|H_{ij}|r_{z,j}+\ell_i$, with $\ell_i$ the residual-disturbance bound.

## Key formulas

- $$\mathcal{Z}=\left\{z\in\mathbb{R}^{N}:\ |z_i|\leq r_{z,i},\ i\in\mathcal{N}\right\},\qquad |y_i(t)|\leq\tilde b_i:=\sum_{j=1}^{N}|(HG)_{ij}|r_{z,j}+\tilde\ell_i.$$
- $$W:=HU,\qquad U_{ij}=0\ \text{for }(i,j)\notin\mathcal{M},\qquad -A\leq W\leq A,\qquad -V\leq U\leq V.$$
- $$r_z\geq |z(0)|,\qquad r_z\geq Db,\qquad r_z\geq\underline r,\qquad -\overline\alpha\odot r_z\leq\operatorname{diag}(W)\leq-\underline\alpha\odot r_z.$$
- $$\dot z=-\operatorname{diag}(\alpha)z+Ux,\quad |x_j(t)|\leq b_j,\qquad \alpha_i r_{z,i}\geq\sum_j|U_{ij}|b_j,\qquad y=Hz+r,\quad |r_i|\leq\ell_i.$$

## Implementation notes

(1) Integration point: implement this as a recurrent residual block between transformer or MLP layers, or as a small diagonal state-space module. Let $x_t\in\mathbb{R}^m$ be a normalized feature disturbance or current layer embedding, let $z_t\in\mathbb{R}^N$ be the bounded latent state, and let $y_t=Hz_t+r_t$ be the feature correction added to the main network. Use a fixed sparse mask $\mathcal{M}$ to determine which state coordinates may communicate. Estimate $b_j$ from a calibration set using a high quantile plus a safety factor, and estimate $\ell_i$ as the maximum absolute residual after fitting $H$. (2) Solve an offline LP. Variables are $U$, $r_z$, and auxiliary nonnegative variables $S_{ij}\geq|U_{ij}|$. Minimize $\sum_i r_{z,i}+\lambda\sum_{ij}S_{ij}$ subject to $-S\leq U\leq S$, $U_{ij}=0$ outside $\mathcal{M}$, $r_{z,i}\geq|z_{0,i}|$, $r_{z,i}\geq\sum_jS_{ij}b_j/\alpha_i$, optional gain bounds $-V\leq U\leq V$, and output constraints $\sum_j|H_{ij}|r_{z,j}+\ell_i\leq\mu_i$. Convert the continuous update to code with Euler step $z_{t+1}=(1-\Delta\alpha)\odot z_t+\Delta Ux_t$, requiring $0<\Delta\alpha_i\leq1$; alternatively use exact diagonal decay $z_{t+1}=e^{-\Delta\alpha}\odot z_t+(1-e^{-\Delta\alpha})\odot(Ux_t/\alpha)$. (3) The LP supplies $U$ and certified radii; empirical calibration supplies $b$, $\ell$, $H$, and the step size. During training, optionally penalize certificate slack $\sum_i\operatorname{ReLU}(|y_i|-\mu_i)$, but do not rely on it for the guarantee. (4) First experiment: compare a 2-layer GRU or diagonal SSM with this module on sequential CIFAR-10, Speech Commands, or a small character language-model task under Gaussian, adversarial, and held-out distribution-shift perturbations. Match parameter count and FLOPs against a standard recurrent residual block. Measure accuracy, hidden-state violation rate, gradient-norm variance, and robustness when test perturbations exceed calibration bounds by 10--50%. The expected signal is near-zero certified violations inside the bound, lower activation and gradient explosions, and equal or better accuracy under perturbation at similar compute.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.