# Åberg Percolation Routing

- ID: 2266
- Canonical URL: https://synthcore.org/idea/2266/berg-percolation-routing
- API JSON: https://synthcore.org/api/idea/2266.json
- API Markdown: https://synthcore.org/api/idea/2266.md
- Verification status: unverified
- Source: [arXiv:2608.13855](https://arxiv.org/abs/2608.13855)
- Category: architecture
- Solves: scalability, stability, accuracy
- ML areas: training-dynamics, transformer, optimizer
- Math tags: statistical-mechanics, graph-theory, dynamical-systems, spectral-theory
- Ratings: usefulness 6/10; difficulty 6/10; novelty 8/10

## Idea description

Replace a dense neural interaction graph by a dynamically activated graph whose edge $(u,v)$ is retained only when its effective coupling exceeds the local spacing of response modes. The network remains sparse below the connectivity transition but becomes globally communicating once a giant component forms, providing a controllable alternative to arbitrary magnitude pruning.

## Mathematical statement

The paper uses a Hamiltonian $H=\sum_i\Gamma_i\sigma_i^z+\sum_{i<j}J_{ij}\sigma_i^x\sigma_j^x$, where $\Gamma_i$ are local fields and $J_{ij}$ are residual couplings. In the computational-state basis, let $E_u$ and $E_v$ be uncoupled energies, $V_{uv}$ the matrix element induced by an interaction, and $\Delta_{uv}$ the local spacing between resonantly accessible states. The Åberg graph retains $(u,v)$ when $|V_{uv}|>\Delta_{uv}$; otherwise it is discarded. For a neural interaction graph, define $V_{uv}$ as the norm of the cross-block Jacobian or attention-logit coupling between units $u$ and $v$, and define $\Delta_{uv}$ as a robust local spacing of the corresponding linearized response spectrum. The useful transition is the emergence of a giant component when the retained mean degree $\bar{k}_{\mathrm{ret}}$ crosses the graph's percolation threshold; for an approximately locally tree-like random graph this occurs near $\bar{k}_{\mathrm{ret}}=1$, while the exact threshold should be estimated for the architecture's graph ensemble.

## Key formulas

- $$H=\sum_i\Gamma_i\sigma_i^z+\sum_{i<j}J_{ij}\sigma_i^x\sigma_j^x,$$
- $$A_{uv}=\mathbf{1}\!\left\{|V_{uv}|>\Delta_{uv}\right\},\qquad V_{uv}=\left\|J_{uv}^{\mathrm{cross}}\right\|_F,$$
- $$\bar{k}_{\mathrm{ret}}=\frac{1}{|\mathcal V|}\sum_{u\in\mathcal V}\sum_{v\neq u}A_{uv},\qquad \text{giant-component onset: }\bar{k}_{\mathrm{ret}}\approx 1\text{ for a locally tree-like random graph},$$
- $$\mathcal L_{\mathrm{route}}=\mathcal L_{\mathrm{task}}+\lambda\max\!\left(0,\,k_{\min}-\bar{k}_{\mathrm{ret}}\right)^2+\mu\frac{1}{|\mathcal E|}\sum_{(u,v)}A_{uv}.$$

## Implementation notes

1. Integration point: apply this to a transformer block, MLP layer, or modular recurrent network. Treat tokens, heads, channels, or modules as graph vertices. Every few hundred optimizer steps, compute a cheap proxy for the interaction matrix. For an MLP, use the Frobenius norm of the block Jacobian $\partial h_v^{out}/\partial h_u^{in}$; for attention, use the average absolute attention-logit derivative or the product of query-key and value-output block norms. Estimate each vertex's local response spacing $\Delta_{uv}$ from the median gap between the largest singular values of its block Jacobian, using an exponential moving average and floor $\epsilon$. Set a binary or straight-through gate $A_{uv}=1[|V_{uv}|>\alpha\Delta_{uv}]$, where $\alpha$ is tunable, and execute only retained edges. 2. Pseudocode: compute block couplings $V$; compute local spacings $\Delta$; form gates $A=(|V|>\alpha\Delta)$; measure connected components and retained mean degree; forward-propagate using only $A$; optionally add the routing penalty above and update $\alpha$ by bisection to target a chosen distance above or below percolation. 3. Computed from the paper: the interaction-versus-spacing rule and giant-component diagnostic. Estimated empirically: neural Jacobian couplings, response spacings, and the architecture-specific threshold. 4. First cheap experiment: train a 6-layer transformer on WikiText-2, comparing dense attention, fixed magnitude pruning at equal FLOPs, and Åberg routing. Sweep $\alpha$ logarithmically. Plot largest-component fraction, average shortest path, gradient norm, loss, and validation accuracy. The quantitative prediction is a sharp connectivity transition when $\bar{k}_{\mathrm{ret}}$ crosses the empirically measured threshold, approximately $1$ for a random-like graph. Just above this point, long-range influence should be preserved at lower edge count; below it, gradients and information should remain confined to finite components. Test whether the transition location is more stable in retained mean degree than in raw $\alpha$.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.