# Differentially Passive Neural Blocks

- ID: 2975
- Canonical URL: https://synthcore.org/idea/2975/differentially-passive-neural-blocks
- API JSON: https://synthcore.org/api/idea/2975.json
- API Markdown: https://synthcore.org/api/idea/2975.md
- Verification status: unverified
- Source: [arXiv:2609.00497](https://arxiv.org/abs/2609.00497)
- Category: architecture
- Solves: stability, generalization, accuracy
- ML areas: mlp, rnn, ssm, training-dynamics
- Math tags: control-theory, dynamical-systems, optimization, linear-algebra, spectral-theory
- Ratings: usefulness 8/10; difficulty 6/10; novelty 6/10

## Idea description

Replace selected residual, recurrent, or state-space blocks by modules whose input-output Jacobians satisfy an IODP inequality throughout a prescribed activation domain. The constraint controls incremental amplification between two trajectories without requiring either trajectory to remain near one fixed equilibrium, so it should improve robustness to changing contexts and prevent exploding long-horizon sensitivities.

## Mathematical statement

For a static neural map y=h(u) with Jacobian J(u)=∂h/∂u, the paper's IODP condition with indices (σ,ρ) is -σI+(J+Jᵀ)/2-ρJᵀJ⪰0 for every u in the certified domain. Here I is the input-dimension identity, σ is the passivity index, and ρ is the output-differential index. For a dynamic neural block ẋ=f(x,u), y=h(x,u), let A=∂f/∂x, B=∂f/∂u, C=∂h/∂x, and D=∂h/∂u. A positive-definite matrix P=Pᵀ≻0 and ε>0 certify the local differential storage V=δxᵀPδx/2 when the paper's LMI holds: [PA+AᵀP+εI, PB; BᵀP, 0] - [0,I; C,D]ᵀ[-σI,I/2; I/2,-ρI][0,I; C,D]⪯0. The first term supplies differential decay through ε; the second supplies incremental input-output dissipation. For a discrete residual block zₖ₊₁=zₖ+ηrθ(zₖ,uₖ), use its sampled Jacobian M=∂zₖ₊₁/∂zₖ and require MᵀPM-P⪯-αP with P≻0 and α>0; this predicts trajectory-distance decay by at least (1-α)ᵏ for identical inputs.

## Key formulas

- $$-\sigma I+\frac{J(u)+J(u)^{\mathsf T}}{2}-\rho J(u)^{\mathsf T}J(u)\succeq0,\qquad J(u)=\frac{\partial h_\theta(u)}{\partial u}.$$
- $$\begin{bmatrix}PA+A^{\mathsf T}P+\epsilon I&PB\\B^{\mathsf T}P&0\end{bmatrix}-\begin{bmatrix}0&I\\C&D\end{bmatrix}^{\mathsf T}\begin{bmatrix}-\sigma I&I/2\\I/2&-\rho I\end{bmatrix}\begin{bmatrix}0&I\\C&D\end{bmatrix}\preceq0.$$
- $$M^{\mathsf T}PM-P\preceq-\alpha P,\qquad M=\frac{\partial z_{k+1}}{\partial z_k},\quad P\succ0,\quad 0<\alpha<1.$$
- $$\|\delta z_k\|_P\le(1-\alpha)^{k/2}\|\delta z_0\|_P,\qquad \|v\|_P^2=v^{\mathsf T}Pv.$$

## Implementation notes

Integration point: begin with an MLP residual block or a small recurrent cell, because its input and state Jacobians can be computed with automatic differentiation. Parameterize P=LLᵀ+δI, where L is learned and δ=10⁻⁴, and choose target indices σ,ρ and margin α>0. For each minibatch, sample actual activations plus perturbed points within the intended domain, compute J by Jacobian-vector products, and form the symmetric violation matrix S=-σI+(J+Jᵀ)/2-ρJᵀJ. Penalize positive eigenvalues using L_IODP=softplus(λmax(-S)+m)², where m is a safety margin. For a residual block, compute the exact Jacobian M of the implemented update, or use M=I+η∂rθ/∂z, then add softplus(λmax(P⁻¹ᐟ²(MᵀPM-P+αP)P⁻¹ᐟ²))². Pseudocode is: sample (z,u); compute J or M; estimate the largest eigenvalue with 5 to 10 power iterations; add the violation penalty to task loss; backpropagate; periodically evaluate dense-domain violations and empirical trajectory-distance ratios. The paper supplies the local matrix inequality and compositional interpretation; the neural implementation estimates worst-case violations from minibatch and adversarially perturbed samples, so certification is empirical unless interval or spectral bounds are added. First cheap experiment: train a 2-layer residual MLP on MNIST classification at depths 10, 50, and 100, comparing unconstrained training with IODP-constrained training. Run identical-input trajectory tests from pairs of perturbed hidden states while sweeping η. The predicted signature is that constrained hidden-distance ratios remain below (1-α)¹ᐟ² per step, while the unconstrained model crosses ratio one when the largest generalized eigenvalue of MᵀPM-P reaches zero. The observed divergence boundary in η should track the LMI boundary within approximately 20 percent; increasing α should lower the stable η ceiling but produce exponential perturbation decay.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.