# Reaction-Closed Sparse Routing

- ID: 227
- Canonical URL: https://synthcore.org/idea/227/reaction-closed-sparse-routing
- API JSON: https://synthcore.org/api/idea/227.json
- API Markdown: https://synthcore.org/api/idea/227.md
- Verification status: unverified
- Source: [arXiv:2608.27201](https://arxiv.org/abs/2608.27201)
- Category: architecture
- Solves: stability, scalability, accuracy
- ML areas: moe-routing, mlp, training-dynamics
- Math tags: combinatorics, graph-theory, dynamical-systems
- Ratings: usefulness 4/10; difficulty 5/10; novelty 7/10

## Idea description

Represent the active experts or channels of a sparse layer by a presence set and impose a reaction-style dependency graph on possible support changes. During a growth phase, activate only the least support set closed under enabled dependencies; during later pruning, allow trajectory-dependent removals but never add structurally unreachable experts. This should reduce routing churn and dead experts while preserving adaptive sparsity.

## Mathematical statement

The paper encodes a reaction network by a stoichiometric matrix $N\in\mathbb{Z}^{n\times m}$, where $N_{ij}$ is the net change of species $s_i$ caused by reaction $r_j$. For a nonnegative continuous state $x\in\mathbb{R}_{\geq 0}^n$, its presence set is $\varphi(x)=\{s_i\in S\mid x_i>0\}$. Define a support rule for each reaction $r_j$ by an input set $I_j$ and produced set $P_j$; a set $A\subseteq S$ is closed when $I_j\subseteq A$ implies $P_j\subseteq A$ for every enabled rule. The least closed superset is computed by $A^{(0)}=A$ and $A^{(k+1)}=A^{(k)}\cup\bigcup_{j:I_j\subseteq A^{(k)}}P_j$ until a fixed point, which is $\operatorname{clos}(A)$. Theorem 3.15 states that a trajectory-induced initial presence set $S_0=\phi(c(0))$ has a feasibly admissible transition $S_0\to\operatorname{clos}(S_0)$, while a later transition from the closure to an organization $O$ is feasible when an actual trajectory realizes it. The neural adaptation treats experts or channels as species, dependency rules as reactions, router logits as kinetic quantities, and active top-k selections as presence sets.

## Key formulas

- $$N\in\mathbb{Z}^{n\times m},\qquad N_{ij}=\text{net change of species }s_i\text{ caused by reaction }r_j.$$
- $$\varphi(x):=\{s_i\in S\mid x_i>0\}.$$
- $$S_0\longrightarrow\operatorname{clos}(S_0),\qquad \phi(c(t_1))=\operatorname{clos}(S_0),\ \phi(c(t_2))=O\ \Longrightarrow\ \operatorname{clos}(S_0)\longrightarrow O\text{ is feasibly admissible}.$$
- $$A^{(0)}=A,\qquad A^{(k+1)}=A^{(k)}\cup\bigcup_{j:I_j\subseteq A^{(k)}}P_j,\qquad \operatorname{clos}(A)=A^{(K)}\ \text{where }A^{(K+1)}=A^{(K)}.$$

## Implementation notes

(1) Integrate this into the router of a sparse Mixture-of-Experts feed-forward layer or into channel gating in a dynamically sparse MLP. Let there be $n$ experts. Store a fixed dependency table containing rules $(I_j,P_j)$; for example, $I_j=\{2,5\}$ and $P_j=\{7\}$ means expert 7 may be activated only after experts 2 and 5 are active. The router computes ordinary logits $z\in\mathbb{R}^n$, but its binary support is constrained by the closure mechanism. (2) For each token, initialize $A$ with the top-$k$ router logits. Repeatedly compute $A_{new}=A\cup\bigcup_{j:I_j\subseteq A}P_j$ until $A_{new}=A$. Score and execute experts in the resulting support, then renormalize gate probabilities over that support. If the closure exceeds the compute budget $k_{max}$, retain the highest-logit members of the closure and mark omitted members as deferred candidates. During warmup, permit closure expansions but disable pruning. Afterwards, prune experts whose exponentially averaged gate mass is below threshold $\tau$, creating a trajectory-dependent downward transition. (3) Compute closure masks exactly from $(I_j,P_j)$; estimate router logits, gate masses, support churn, and pruning thresholds empirically. The dependency graph can be manually specified from layer hierarchy or sampled as a sparse DAG. (4) First experiment: train a 12-layer, 32-expert Switch-style Transformer on WikiText-103 or C4 with $k=2$. Compare ordinary top-k routing, load-balanced top-k routing, and reaction-closed routing at equal activated-expert FLOPs. Measure validation perplexity, permanently dead experts, adjacent-batch support Jaccard churn, token overflow rate, closure overhead, and wall-clock throughput. The hypothesis is fewer dead experts and lower support churn during warmup at equal perplexity or improved perplexity; failure is indicated by increased routing overhead or worse perplexity at fixed FLOPs. A cheaper pilot uses a two-layer sparse MLP on CIFAR-100 with a manually defined channel hierarchy.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.