# Gain-Weighted Cluster Co-Design

- ID: 1042
- Canonical URL: https://synthcore.org/idea/1042/gain-weighted-cluster-co-design
- API JSON: https://synthcore.org/api/idea/1042.json
- API Markdown: https://synthcore.org/api/idea/1042.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.16985](https://arxiv.org/abs/2607.16985)
- Category: architecture
- Solves: stability, scalability
- ML areas: optimizer, transformer, moe-routing, regularization, training-dynamics
- Math tags: control-theory, dynamical-systems, linear-algebra, graph-theory, optimization
- Ratings: usefulness 7/10; difficulty 7/10; novelty 7/10

## Idea description

Use small-gain diagnostics to jointly learn module normalization and a communication partition rather than imposing a fixed global spectral constraint. Clusters should be formed around high-gain feedback loops, because grouping weakly related modules cannot improve the certificate and only adds bookkeeping.

## Mathematical statement

Let Gamma be the nonnegative matrix of measured inter-module gains and let d_i>0 be normalization weights. Diagonal scaling produces the normalized gain matrix Gamma_d=D^{-1}Gamma D, where D=diag(d_1,...,d_n). Its worst local certificate is mu(d)=max_i sum_j gamma_ij d_j/d_i. For a partition P of nodes into clusters S_a, define g_ab as the maximum product of gains over paths from cluster S_b to cluster S_a. A partition can tighten a decentralized certificate only when the problematic high-gain channels are internalized; otherwise the same failing cycle remains visible at the cluster level. The weights d_i are parameterized as d_i=exp(q_i), and q_i are optimized jointly or alternately with the neural parameters.

## Key formulas

- $$D=\operatorname{diag}(d_1,\ldots,d_n),\qquad \Gamma_d=D^{-1}\Gamma D,\qquad \mu(d)=\|\Gamma_d\|_{\infty}=\max_i\sum_j\gamma_{ij}\frac{d_j}{d_i},$$
- $$g_{ab}:=\max_{p:S_b\leadsto S_a}\prod_{(j\to i)\in p}\gamma_{ij},\qquad \text{require }g_{ab}g_{ba}<1\text{ for every two-cluster feedback path pair},$$
- $$\mathcal{L}_{\mathrm{co-design}}=\mathcal{L}_{\mathrm{task}}+\lambda\,\operatorname{softplus}(\mu(e^q)-1+\varepsilon)+\beta\sum_{a\neq b}g_{ab}+\tau\,\mathrm{Cut}(P),$$
- $$s_{ij}=\log(\gamma_{ij}+\delta)+\log(\gamma_{ji}+\delta),\qquad (i,j)_{\mathrm{merge}}=\arg\max_{i,j}s_{ij}$$

## Implementation notes

Integrate this method into a block-sparse transformer, mixture-of-experts router, or modular recurrent model. Begin with a fixed number of modules and expose the communication pattern through an adjacency mask. Every K training steps, estimate Gamma from block JVPs on a calibration minibatch. Parameterize normalization as d_i=exp(q_i), take five to twenty gradient steps on q to reduce mu(exp(q)), and then greedily merge the pair of modules with the largest symmetric feedback score s_ij=log(gamma_ij+delta)+log(gamma_ji+delta), subject to a maximum cluster size. Within each cluster enumerate cycles up to length four; across clusters compute maximum-gain paths using dynamic programming on log gains, capped at path length L. During ordinary training add the displayed barrier and boundary penalties, but recompute the graph less frequently than model weights. Quantities computed from the paper are the diagonal gain normalization, cycle/path conditions, and the requirement that a partition must internalize failure-causing channels; quantities estimated empirically are JVP gains and path maxima. First cheap experiment: train a six-block residual MLP on CIFAR-10 with deliberately amplified cross-block residual couplings, comparing fixed singleton clusters, random clusters, and gain-weighted greedy clusters. Sweep coupling strength c. The prediction is that the singleton certificate fails when the worst cycle product exceeds 1, while a useful partition improves the certified margin only after every edge in that dominant cycle lies within one cluster. Random partitions should not show this transition.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a self-contained gain-weighted cluster co-design toy implementation covering diagonal gain normalization, symmetric feedback-score greedy merging, and dominant-cycle visibility. The math checks passed: optimized mu matched the predicted cycle gain with maximum absolute error 1.18e-8, imbalance invariance error was 9.27e-9, and the c=1 stability boundary plus clustering transition were confirmed across sweeps. This is a real mechanism-level effect, but not evidence of improved neural task loss or runtime.

### Mechanism check

- Verdict: Built a self-contained gain-weighted cluster co-design toy implementation covering diagonal gain normalization, symmetric feedback-score greedy merging, and dominant-cycle visibility. The math checks passed: optimized mu matched the predicted cycle gain with maximum absolute error 1.18e-8, imbalance invariance error was 9.27e-9, and the c=1 stability boundary plus clustering transition were confirmed across sweeps. This is a real mechanism-level effect, but not evidence of improved neural task loss or runtime.
- Confidence: 9/10
- Limitations: No block-sparse transformer, recurrent model, CIFAR-10 training, JVP gain estimation, learned q during neural optimization, path dynamic programming, communication cost, wall-clock speed, memory, or task-loss comparison was tested. The cluster result is a graph-level certificate/visibility demonstration and does not reduce the full spectral radius of the original graph.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.778
- Paired wins: 4/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/424/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/424/bench_report.json)
- [experiment.py](https://synthcore.org/code/424/experiment.py)
- [report.md](https://synthcore.org/code/424/report.md)
- [report_bench_2026-08-31T120247.md](https://synthcore.org/code/424/report_bench_2026-08-31T120247.md)
- [results.json](https://synthcore.org/code/424/results.json)
- [Download all files as ZIP](https://synthcore.org/download/424)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.