Differentially Passive Neural Blocks

Integration point: begin with an MLP residual block or a small recurrent cell, because its input and state Jacobians can be computed with automatic differentiation. Parameterize P=LLᵀ+δI, where L is learned and δ=10⁻⁴, and choose target indices σ,ρ and margin α>0. For each minibatch, sample actual activations plus perturbed points within the intended domain, compute J by Jacobian-vector products, and form the symmetric violation matrix S=-σI+(J+Jᵀ)/2-ρJᵀJ. Penalize positive eigenvalues using L_IODP=softplus(λmax(-S)+m)², where m is a safety margin. For a residual block, compute the exact Jacobian M of the implemented update, or use M=I+η∂rθ/∂z, then add softplus(λmax(P⁻¹ᐟ²(MᵀPM-P+αP)P⁻¹ᐟ²))². Pseudocode is: sample (z,u); compute J or M; estimate the largest eigenvalue with 5 to 10 power iterations; add the violation penalty to task loss; backpropagate; periodically evaluate dense-domain violations and empirical trajectory-distance ratios. The paper supplies the local matrix inequality and compositional interpretation; the neural implementation estimates worst-case violations from minibatch and adversarially perturbed samples, so certification is empirical unless interval or spectral bounds are added. First cheap experiment: train a 2-layer residual MLP on MNIST classification at depths 10, 50, and 100, comparing unconstrained training with IODP-constrained training. Run identical-input trajectory tests from pairs of perturbed hidden states while sweeping η. The predicted signature is that constrained hidden-distance ratios remain below (1-α)¹ᐟ² per step, while the unconstrained model crosses ratio one when the largest generalized eigenvalue of MᵀPM-P reaches zero. The observed divergence boundary in η should track the LMI boundary within approximately 20 percent; increasing α should lower the stable η ceiling but produce exponential perturbation decay.