# Clustered Small-Gain Certificate for Modular Neural Dynamics

- ID: 1041
- Canonical URL: https://synthcore.org/idea/1041/clustered-small-gain-certificate-for-modular-neural-dynamics
- API JSON: https://synthcore.org/api/idea/1041.json
- API Markdown: https://synthcore.org/api/idea/1041.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.16985](https://arxiv.org/abs/2607.16985)
- Category: dynamics
- Solves: stability, scalability
- ML areas: optimizer, rnn, transformer, training-dynamics
- Math tags: control-theory, dynamical-systems, linear-algebra, graph-theory, optimization
- Ratings: usefulness 8/10; difficulty 6/10; novelty 6/10

## Idea description

Treat neural modules as interconnected dynamical subsystems and estimate the gain from every module input to every neighboring module output. Replace an expensive global Jacobian spectral-radius calculation by decentralized directed-cycle tests inside clusters and path-gain tests between clusters. Penalizing violations during training should prevent exploding recurrent trajectories while retaining less conservative behavior than constraining every individual block independently.

## Mathematical statement

Let modules be indexed by nodes i in V, with perturbation dynamics z_i and stable local input-output maps whose induced gain from module j to module i is gamma_ij >= 0. The gain graph has an edge j -> i when gamma_ij > 0. For a directed cycle C, the cyclic small-gain condition is the product of gamma_ij over all edges in C being less than one. A sufficient global certificate is obtained by finding positive weights d_i such that the scaled gain matrix has maximum row sum below one: mu(d)=max_i d_i^{-1} sum_j gamma_ij d_j < 1. Here gamma_ij is estimated from block Jacobians or empirical perturbation responses, d_i are positive scaling weights, and 1-mu(d) is the certified contraction reserve.

## Key formulas

- $$z_i^{+}=f_i(z_i,u_i),\qquad u_i=\sum_{j\neq i}G_{ij}z_j,\qquad \|\delta y_i\|\leq\sum_j\gamma_{ij}\|\delta y_j\|,$$
- $$\Gamma=[\gamma_{ij}],\qquad \prod_{(j\rightarrow i)\in C}\gamma_{ij}<1\quad\text{for every directed feedback cycle }C,$$
- $$\mu(d):=\max_i\frac{1}{d_i}\sum_j\gamma_{ij}d_j<1,\qquad d_i>0,$$
- $$\mathcal{L}_{\mathrm{sg}}=\sum_{C}\operatorname{softplus}\!\left(\log\prod_{e\in C}\gamma_e-\log(1-\varepsilon)\right)$$

## Implementation notes

Use a modular recurrent network, residual network, or transformer in which each block is a subsystem and cross-block communication is an edge. At a minibatch point, compute block Jacobian-vector products J_ij v_j with automatic differentiation, using two to four Hutchinson or power-iteration vectors per source block. Estimate gamma_ij as a high quantile of ||J_ij v_j||/||v_j||, optionally multiplied by a safety factor of 1.2. Construct the sparse nonnegative gain matrix Gamma. Enumerate short directed cycles within each selected cluster; for longer graphs, approximate the worst cycle using log-gain search or optimize positive weights d with gradient descent on mu(d). Add the cycle barrier to the task loss. Pseudocode: sample perturbation vectors; obtain block JVPs; update gamma_ij with an exponential moving maximum; compute cycle products and scaled row sums; add the certificate penalty; perform the ordinary optimizer step. The paper-derived quantities are cycle and path small-gain inequalities; empirical quantities are Jacobian gains, clustering, and d. First experiment: compare a four- to eight-block GRU or residual MLP on sequential MNIST and a synthetic long-horizon copy task against spectral normalization and an unconstrained baseline. Sweep the penalty coefficient and measure trajectory norms over 1,000 steps. The prediction is a sharp divergence boundary: when the estimated worst cycle product crosses 1, long-horizon perturbations should stop decaying and begin growing. Certified runs should have empirical per-step amplification below 1, with the measured boundary agreeing with the gain estimate within approximately 20%.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built and numerically verified a clustered small-gain certificate for modular dynamics. The symmetric-cycle boundary was observed at q=1.00 versus the predicted 1.00; in the 4-node sweep, positive trajectory growth first appeared at scale 1.05 versus predicted spectral-radius scale 1.012 (3.7% grid/tolerance error). Path gain scaling matched log(q) to machine precision, and optimized positive scaling exactly recovered the 2-cycle spectral radius while unscaled row sums were more conservative. At an unstable scale, clustered attenuation reduced rho to 0.907 and final norm to 7.5e-6, versus rho 0.991 and final norm 0.249 for independent edge clipping, while changing parameters much less on average (0.0152 vs 0.3875 in the analytic comparison).

### Mechanism check

- Verdict: Built and numerically verified a clustered small-gain certificate for modular dynamics. The symmetric-cycle boundary was observed at q=1.00 versus the predicted 1.00; in the 4-node sweep, positive trajectory growth first appeared at scale 1.05 versus predicted spectral-radius scale 1.012 (3.7% grid/tolerance error). Path gain scaling matched log(q) to machine precision, and optimized positive scaling exactly recovered the 2-cycle spectral radius while unscaled row sums were more conservative. At an unstable scale, clustered attenuation reduced rho to 0.907 and final norm to 7.5e-6, versus rho 0.991 and final norm 0.249 for independent edge clipping, while changing parameters much less on average (0.0152 vs 0.3875 in the analytic comparison).
- Confidence: 8/10
- Limitations: The main experiment uses a linear 4-module system with exact absolute Jacobian gains, not a trained GRU/residual MLP. It does not implement minibatch autograd JVP estimation, EMA gain tracking, cycle enumeration for arbitrary sparse graphs, task-loss optimization, FLOP accounting, or sequential-MNIST/copy-task generalization. The clustered intervention uses common attenuation rather than fully learning per-module scaling weights.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.01485
- Paired wins: 1/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/184/bench_report.json)
- [clustered_small_gain.py](https://synthcore.org/code/184/clustered_small_gain.py)
- [report.md](https://synthcore.org/code/184/report.md)
- [report_bench_2026-08-30T194252.md](https://synthcore.org/code/184/report_bench_2026-08-30T194252.md)
- [run_experiment.py](https://synthcore.org/code/184/run_experiment.py)
- [stage2_clustered_gain.py](https://synthcore.org/code/184/stage2_clustered_gain.py)
- [Download all files as ZIP](https://synthcore.org/download/184)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.