Small-Gain Constrained Neural Modules

Use this at the boundary between residual blocks, recurrent modules, or MoE experts rather than on every individual scalar neuron. Write the model as x_i^+=f_i(x_i,u_i) and expose explicit couplings u_i=sum_j c_ij h_j(x_j); for a transformer, modules can be attention, MLP, normalization, and residual streams, while for an RNN they can be state-update and feedback branches. At each training checkpoint, estimate a_i and b_i by power iteration or minibatch Jacobian-vector products over a prescribed activation region: estimate a_i approximately as the maximum of ||partial f_i/partial x_i||_2 and b_i as the maximum of ||partial f_i/partial u_i||_2 over random and high-norm activation samples. Estimate r_j similarly for h_j, construct G, and compute its spectral radius by eigendecomposition. Add the penalty lambda max(0,rho(G)-(1-epsilon))^2 to the task loss, or rescale coupling weights whenever rho(G)>1-epsilon. Pseudocode: compute local gains; form G; calculate rho(G); backpropagate task loss plus gain penalty; optionally project edge weights by c_ij <- c_ij(1-epsilon)/rho(G) when the boundary is violated. Separately fit P positive definite from empirical covariance or solve a discrete Lyapunov inequality for a linearized module graph, then test the ellipsoid contraction ratio. The paper contributes the small-gain assembly and invariant-region logic; Jacobian norms, gain estimation, and projection are engineering adaptations. The first cheap experiment should use a 4-block residual MLP on CIFAR-10 or a delayed-copy sequence task, comparing unconstrained residual training with the spectral-radius penalty at equal parameter count. Sweep a global coupling multiplier alpha and predict a sharp transition near alpha_c=1/rho(G_0), where G_0 is the unscaled measured gain matrix: activation perturbations and long-horizon Jacobian norms should decay for alpha<alpha_c and grow for alpha>alpha_c, with the observed boundary within 20% of prediction. Also test whether the estimated ellipsoid contraction ratio remains below one for perturbations inside E_r and crosses one as r increases beyond the certified region.