{
 "artifacts": null,
 "category": "architecture",
 "description": "Replace an unconstrained attention or MoE routing matrix with a row-stochastic matrix that preserves a learned Gibbs-like distribution over groups. Anneal its temperature so that the router converges to a block-upper-triangular operator: mixing remains flexible within equal-energy groups, while cross-group traffic becomes directional rather than oscillating or collapsing. This creates a hierarchy-aware inductive bias and can reduce the number of active expert or attention connections.",
 "formulas_latex": [
  "$$\\gamma_i(\\beta)=\\frac{e^{-\\beta E_i}}{Z(\\beta)},\\qquad Z(\\beta)=\\sum_{k=1}^{n}e^{-\\beta E_k}.$$",
  "$$P_{ij}\\ge 0,\\qquad \\sum_{j=1}^{n}P_{ij}=1,\\qquad P\\gamma(\\beta)=\\gamma(\\beta).$$",
  "$$\\beta_* = \\frac{\\log(2)}{E_1-E_3},\\qquad \\gamma=(\\gamma_1,\\gamma_1,\\gamma_3)^T,$$",
  "$$A^{(2,1)}_{11}=\\begin{pmatrix}1-\\frac{2\\gamma_1}{\\gamma_3}\u0026\\frac{\\gamma_1}{\\gamma_3}\u0026\\frac{\\gamma_1}{\\gamma_3}\\\\1\u00260\u00260\\\\1\u00260\u00260\\end{pmatrix},\\qquad \\lim_{\\beta\\to\\infty}\\gamma_1=0,\\quad \\lim_{\\beta\\to\\infty}\\gamma_3=1.$$"
 ],
 "id": 3162,
 "implementation": "Integrate this at the attention-probability or MoE-router tensor, after computing ordinary logits and before multiplying by values or dispatching tokens. Partition the \\(m\\) tokens, experts, or channels into \\(K\\) ordered groups; assign each group a learned energy \\(E_g\\), and define \\(\\gamma_g=\\exp(-\\beta E_g)/\\sum_h\\exp(-\\beta E_h)\\). Build a nonnegative unconstrained matrix \\(Q\\) from router logits. Parameterize the final matrix with an upper-block mask so entries from a later group into an earlier group are zero, then project it toward the Gibbs-preserving equality \\(P\\gamma=\\gamma\\). For an MVP, initialize \\(P=\\operatorname{softmax}(Q,\\mathrm{dim}=1)\\), alternately normalize rows and apply a multiplicative correction based on the ratio \\(\\gamma/(P\\gamma)\\) for 5--10 iterations, followed by row renormalization. Use \\(\\beta(t)=\\beta_{\\min}+ (\\beta_{\\max}-\\beta_{\\min})t/T\\), choosing \\(\\beta_{\\max}\\) so the smallest group weight is approximately \\(10^{-3}\\). Compute \\(Y=PV\\) for attention or dispatch tokens using \\(P\\) for MoE. The paper supplies the Gibbs weights, invariance constraint, critical-temperature intuition, and extremal block structure; group energies and logits are learned empirically. Start with a 6-layer, 8-head Transformer on CIFAR-10 patch classification or WikiText-2, using three token or expert groups. Compare dense softmax attention or routing at equal parameter count and batch size. Record validation accuracy, routing entropy, \\(\\|P\\gamma-\\gamma\\|_1\\), lower-block mass, active-edge count, and wall-clock/FLOPs. Success means equal-or-better accuracy with reduced cross-group traffic and no loss spike during annealing.",
 "math_summary": "The paper studies stochastic matrices preserving a probability vector \\(\\gamma(\\beta)\\), with Gibbs weights \\(\\gamma_i(\\beta)=e^{-\\beta E_i}/Z(\\beta)\\), where \\(E_i\\) is the energy of state \\(i\\), \\(\\beta\\) is inverse temperature, and \\(Z(\\beta)=\\sum_i e^{-\\beta E_i}\\). A matrix \\(P\\) is row-stochastic when \\(P_{ij}\\ge 0\\) and \\(\\sum_jP_{ij}=1\\); it preserves \\(\\gamma\\) under column-vector dynamics when \\(P\\gamma=\\gamma\\). In the doubly-degenerate three-state example \\(E_1=E_2\u003eE_3\\), the critical inverse temperature is \\(\\beta_* = \\log(2)/(E_1-E_3)\\). One finite-temperature extremal operator is \\(A^{(2,1)}_{11}=\\begin{pmatrix}1-2r\u0026r\u0026r\\\\1\u00260\u00260\\\\1\u00260\u00260\\end{pmatrix}\\), with \\(r=\\gamma_1/\\gamma_3\\); it is row-stochastic and preserves \\(\\gamma=(\\gamma_1,\\gamma_1,\\gamma_3)^T\\). As \\(\\beta\\to\\infty\\), the excited-state weight vanishes and the allowed semigroup becomes block-upper-triangular with additional substochasticity constraints. The adaptation uses the same invariance equation and annealed block structure, but applies the resulting matrix to attention values or routing probabilities.",
 "math_tags": [
  "probability",
  "convex-analysis",
  "combinatorics"
 ],
 "ml_areas": [
  "attention",
  "moe-routing",
  "transformer"
 ],
 "paper": {
  "arxiv_id": "2609.04057",
  "arxiv_url": "https://arxiv.org/abs/2609.04057",
  "summary_what_math_gives_to_ml": "The paper gives a constructive description of stochastic maps that preserve a Gibbs distribution when inverse temperature tends to infinity. Its key transferable asset is that this limit is not obtained by simply substituting the limiting distribution into the finite-temperature constraint: it produces block-upper-triangular stochastic operators with substochastic cross-block flow and a finite extremal decomposition. This suggests an energy- or hierarchy-aware neural mixing layer whose transition matrix preserves a prescribed Gibbs-like distribution while annealing toward one-way information flow between learned groups. The most practical first test is a constrained attention or MoE router, compared with ordinary softmax routing at equal compute.",
  "title": "Limits of Stochastic Semigroups and Block-Triangular Majorisation",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 6,
  "novelty": 7,
  "usefulness": 6
 },
 "solves": [
  "stability",
  "generalization",
  "scalability"
 ],
 "title": "Zero-Temperature Block-Triangular Routing",
 "url": "https://synthcore.org/idea/3162/zero-temperature-block-triangular-routing",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}