# Zero-Temperature Block-Triangular Routing

- ID: 3162
- Canonical URL: https://synthcore.org/idea/3162/zero-temperature-block-triangular-routing
- API JSON: https://synthcore.org/api/idea/3162.json
- API Markdown: https://synthcore.org/api/idea/3162.md
- Verification status: unverified
- Source: [arXiv:2609.04057](https://arxiv.org/abs/2609.04057)
- Category: architecture
- Solves: stability, generalization, scalability
- ML areas: attention, moe-routing, transformer
- Math tags: probability, convex-analysis, combinatorics
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Replace an unconstrained attention or MoE routing matrix with a row-stochastic matrix that preserves a learned Gibbs-like distribution over groups. Anneal its temperature so that the router converges to a block-upper-triangular operator: mixing remains flexible within equal-energy groups, while cross-group traffic becomes directional rather than oscillating or collapsing. This creates a hierarchy-aware inductive bias and can reduce the number of active expert or attention connections.

## Mathematical statement

The paper studies stochastic matrices preserving a probability vector \(\gamma(\beta)\), with Gibbs weights \(\gamma_i(\beta)=e^{-\beta E_i}/Z(\beta)\), where \(E_i\) is the energy of state \(i\), \(\beta\) is inverse temperature, and \(Z(\beta)=\sum_i e^{-\beta E_i}\). A matrix \(P\) is row-stochastic when \(P_{ij}\ge 0\) and \(\sum_jP_{ij}=1\); it preserves \(\gamma\) under column-vector dynamics when \(P\gamma=\gamma\). In the doubly-degenerate three-state example \(E_1=E_2>E_3\), the critical inverse temperature is \(\beta_* = \log(2)/(E_1-E_3)\). One finite-temperature extremal operator is \(A^{(2,1)}_{11}=\begin{pmatrix}1-2r&r&r\\1&0&0\\1&0&0\end{pmatrix}\), with \(r=\gamma_1/\gamma_3\); it is row-stochastic and preserves \(\gamma=(\gamma_1,\gamma_1,\gamma_3)^T\). As \(\beta\to\infty\), the excited-state weight vanishes and the allowed semigroup becomes block-upper-triangular with additional substochasticity constraints. The adaptation uses the same invariance equation and annealed block structure, but applies the resulting matrix to attention values or routing probabilities.

## Key formulas

- $$\gamma_i(\beta)=\frac{e^{-\beta E_i}}{Z(\beta)},\qquad Z(\beta)=\sum_{k=1}^{n}e^{-\beta E_k}.$$
- $$P_{ij}\ge 0,\qquad \sum_{j=1}^{n}P_{ij}=1,\qquad P\gamma(\beta)=\gamma(\beta).$$
- $$\beta_* = \frac{\log(2)}{E_1-E_3},\qquad \gamma=(\gamma_1,\gamma_1,\gamma_3)^T,$$
- $$A^{(2,1)}_{11}=\begin{pmatrix}1-\frac{2\gamma_1}{\gamma_3}&\frac{\gamma_1}{\gamma_3}&\frac{\gamma_1}{\gamma_3}\\1&0&0\\1&0&0\end{pmatrix},\qquad \lim_{\beta\to\infty}\gamma_1=0,\quad \lim_{\beta\to\infty}\gamma_3=1.$$

## Implementation notes

Integrate this at the attention-probability or MoE-router tensor, after computing ordinary logits and before multiplying by values or dispatching tokens. Partition the \(m\) tokens, experts, or channels into \(K\) ordered groups; assign each group a learned energy \(E_g\), and define \(\gamma_g=\exp(-\beta E_g)/\sum_h\exp(-\beta E_h)\). Build a nonnegative unconstrained matrix \(Q\) from router logits. Parameterize the final matrix with an upper-block mask so entries from a later group into an earlier group are zero, then project it toward the Gibbs-preserving equality \(P\gamma=\gamma\). For an MVP, initialize \(P=\operatorname{softmax}(Q,\mathrm{dim}=1)\), alternately normalize rows and apply a multiplicative correction based on the ratio \(\gamma/(P\gamma)\) for 5--10 iterations, followed by row renormalization. Use \(\beta(t)=\beta_{\min}+ (\beta_{\max}-\beta_{\min})t/T\), choosing \(\beta_{\max}\) so the smallest group weight is approximately \(10^{-3}\). Compute \(Y=PV\) for attention or dispatch tokens using \(P\) for MoE. The paper supplies the Gibbs weights, invariance constraint, critical-temperature intuition, and extremal block structure; group energies and logits are learned empirically. Start with a 6-layer, 8-head Transformer on CIFAR-10 patch classification or WikiText-2, using three token or expert groups. Compare dense softmax attention or routing at equal parameter count and batch size. Record validation accuracy, routing entropy, \(\|P\gamma-\gamma\|_1\), lower-block mass, active-edge count, and wall-clock/FLOPs. Success means equal-or-better accuracy with reduced cross-group traffic and no loss spike during annealing.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.