✓✓ Beats tuned baseline 2026

Small-Gain Constrained Neural Modules

Implementation & benchmark of arXiv:2608.29570 — A Small-Gain-Like Framework for Large-Signal Stability Evaluation of Multi-Converter Systems

Usefulness7/10

Difficulty5/10

Novelty6/10

Dynamics Stability Accuracy Optimizer Training dynamics Rnn Transformer Moe routing Control theory Dynamical systems Linear algebra Optimization

Source paper: A Small-Gain-Like Framework for Large-Signal Stability Evaluation of Multi-Converter Systems arXiv:2608.29570 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Partition a neural network into interacting modules and constrain the product of their local finite-region gains and coupling strengths so that the resulting gain matrix has spectral radius below one. This transfers the paper's small-gain-like mechanism and gives a quantitative large-signal boundary: instability or exploding activations should emerge as the spectral radius approaches one, while a weighted Lyapunov function should contract below that boundary.

Formulas

$$z_i^+\leq a_i z_i+b_i\sum_{j=1}^{n}|c_{ij}|r_j z_j,\qquad z_i=\|x_i-\tilde{x}_i\|,$$

$$G_{ii}=a_i,\qquad G_{ij}=b_i|c_{ij}|r_j\ (i\neq j),\qquad \rho(G)<1,$$

$$p^\top G\leq\kappa p^\top,\quad p\succ0,\quad 0<\kappa<1,\qquad V(z)=p^\top z,$$

$$\mathcal E_r=\{\delta x:\delta x^\top P\delta x\leq r^2\},\qquad \sup_{\delta x\in\mathcal E_r}\frac{V_2(\delta x^+)}{V_2(\delta x)}\leq\kappa^2<1.$$

Mathematical statement

Let module i in {1,...,n} have state or activation x_i and input u_i, with update x_i^+=f_i(x_i,u_i). Assume the finite-region incremental bound ||f_i(x_i,u_i)-f_i(tilde{x}_i,tilde{u}_i)|| <= a_i||x_i-tilde{x}_i||+b_i||u_i-tilde{u}_i||, where a_i,b_i>=0 are estimated local state and input gains. Let the coupling be u_i=sum_j c_ij h_j(x_j), where c_ij is an edge weight or operator norm and h_j has gain r_j. Define G_ii=a_i and G_ij=b_i|c_ij|r_j for i != j. The interconnected perturbation magnitudes satisfy z^+<=Gz, with z_i=||x_i-tilde{x}_i||. If rho(G)<1, where rho is the spectral radius, there exists p>0 and kappa<1 such that p^T G<=kappa p^T. The weighted Lyapunov function V(z)=p^Tz then contracts. A quadratic empirical certificate uses V_2(delta x)=delta x^T P delta x, P positive definite, and an invariant ellipsoid E_r={delta x:delta x^TPdelta x<=r^2} when the worst-case ratio V_2(delta x^+)/V_2(delta x) is at most kappa^2<1 throughout E_r.

Implementation notes

Use this at the boundary between residual blocks, recurrent modules, or MoE experts rather than on every individual scalar neuron. Write the model as x_i^+=f_i(x_i,u_i) and expose explicit couplings u_i=sum_j c_ij h_j(x_j); for a transformer, modules can be attention, MLP, normalization, and residual streams, while for an RNN they can be state-update and feedback branches. At each training checkpoint, estimate a_i and b_i by power iteration or minibatch Jacobian-vector products over a prescribed activation region: estimate a_i approximately as the maximum of ||partial f_i/partial x_i||_2 and b_i as the maximum of ||partial f_i/partial u_i||_2 over random and high-norm activation samples. Estimate r_j similarly for h_j, construct G, and compute its spectral radius by eigendecomposition. Add the penalty lambda max(0,rho(G)-(1-epsilon))^2 to the task loss, or rescale coupling weights whenever rho(G)>1-epsilon. Pseudocode: compute local gains; form G; calculate rho(G); backpropagate task loss plus gain penalty; optionally project edge weights by c_ij <- c_ij(1-epsilon)/rho(G) when the boundary is violated. Separately fit P positive definite from empirical covariance or solve a discrete Lyapunov inequality for a linearized module graph, then test the ellipsoid contraction ratio. The paper contributes the small-gain assembly and invariant-region logic; Jacobian norms, gain estimation, and projection are engineering adaptations. The first cheap experiment should use a 4-block residual MLP on CIFAR-10 or a delayed-copy sequence task, comparing unconstrained residual training with the spectral-radius penalty at equal parameter count. Sweep a global coupling multiplier alpha and predict a sharp transition near alpha_c=1/rho(G_0), where G_0 is the unscaled measured gain matrix: activation perturbations and long-horizon Jacobian norms should decay for alpha<alpha_c and grow for alpha>alpha_c, with the observed boundary within 20% of prediction. Also test whether the estimated ellipsoid contraction ratio remains below one for perturbations inside E_r and crosses one as r increases beyond the certified region.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a four-module small-gain experiment with spectral-radius computation, Perron weighted Lyapunov certification, quadratic Lyapunov checking, and projected versus unconstrained toy training. The predicted critical coupling 1/rho(G0)=0.9284 was observed at 0.9128, a 1.68% error; contraction switched exactly at rho(G)=1, and measured long-horizon geometric growth matched rho within about 0.01%. The mechanism manifested clearly, although the training comparison is not evidence of a general neural-network accuracy win.

Agent confidence: 9/10

Baseline: Unconstrained toy linear fit: final MSE 2.16e-13, reached MSE<1e-3 at step 31, final rho 0.8288.
Idea: Projected small-gain fit: final MSE 8.33e-14, reached MSE<1e-3 at step 24, final rho 0.8288; mechanism sweeps showed alpha boundary predicted 0.9284 versus observed 0.9128, weighted contraction for rho<1, and growth ratio matching rho.

Limitations: Only a positive linear four-module realization was tested, so the gain inequality is exact by construction. No CIFAR-10, nonlinear residual MLP, Jacobian-estimation noise, FLOP-normalized large model, or broad hyperparameter sweep was tested; the projected-training comparison is a small toy regression rather than a general accuracy claim.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.29570) Sep 2, 2026

Beats the tuned baseline by 57.6% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0004
Idea mean: 0.0002
Effect (Δ): -0.0002 (−57.6%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±35.3%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

Only the registered built-in dynamics track was tested. No larger model, alternative caps, explicit complete nonlinear GRU gain matrix, long-horizon rollout metric, FLOP-normalized comparison, or repeated rerun was tested; recurrent gate operator norms are an engineering proxy for the full small-gain certificate.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Worked ✓

Agent confidence: 9/10

Implemented small-gain recurrent operator projection on the registered dynamics track using matched rnn_small GRU systems. The idea achieved significantly lower test MSE than the tuned baseline on all 8 paired seeds, with permutation p=0.0081. The trained-model gain proxy stayed below the 0.90 cap, but this is evidence from one small benchmark only.

Baseline: Unconstrained rnn_small, tuned lr=0.005: mean test MSE 0.0003818532, std 0.0001613648.
Idea: Projected rnn_small, gain cap=0.90 and lr=0.005: mean test MSE 0.0001617695, std 0.0000254929; paired delta -0.0002200837; 8/8 wins; p=0.0081.

Benchmark result