Unverified 2026

Discrete-Stable Routing Renormalization

Implementation & benchmark of arXiv:2607.18968 — Discrete distributions and statistical mechanics of small systems

Usefulness6/10

Difficulty6/10

Novelty8/10

Regularization Stability Scalability Generalization Moe routing Regularization Training dynamics Optimizer Renormalization Probability Statistical mechanics Stochastic processes

Source paper: Discrete distributions and statistical mechanics of small systems arXiv:2607.18968 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add a scale-consistency regularizer to stochastic MoE or adaptive-computation routing counts. The router is trained so that aggregating independently routed microbatches produces the same normalized count law predicted by the discrete-stable renormalization fixed point, reducing sensitivity to batch size and stream length while allowing heavy-tailed but controlled expert demand.

Formulas

$$G(z)=\mathbb{E}[z^N],\qquad G_{\mathrm{DS}}(z)=\exp\!\left[-c(1-z)^\alpha\right],\quad c>0,\;0<\alpha\leq1.$$

$$(\mathcal{T}_mG)(z)=\left[G\left(1-m^{-1/\alpha}+m^{-1/\alpha}z\right)\right]^m.$$

$$\mathcal{T}_mG_{\mathrm{DS}}(z)=\exp\!\left[-cm\left(m^{-1/\alpha}(1-z)\right)^\alpha\right]=G_{\mathrm{DS}}(z).$$

$$\mathcal{L}_{\mathrm{RG}}=\sum_{e=1}^{E}\sum_{m\in\mathcal{M}}\sum_{z\in\mathcal{Z}}\left|\widehat G_{e,m}(z)-\left[\widehat G_{e,1}\left(1-m^{-1/\alpha}+m^{-1/\alpha}z\right)\right]^m\right|^2.$$

Mathematical statement

For a nonnegative integer random variable N, define its probability generating function by G(z)=E[z^N]. A discrete-stable fixed point has G_DS(z)=exp[-c(1-z)^alpha], where c>0 is a scale parameter and 0<alpha<=1 controls aggregation and tail behavior. Let T_m act on m independent copies by thinning each count with probability q=m^(-1/alpha) and summing the retained counts. Thinning changes the generating function from G(z) to G(1-q+qz), while summing m independent variables raises the result to the m-th power. Therefore (T_m G)(z)=[G(1-q+qz)]^m. Substituting the discrete-stable form gives T_m G_DS=G_DS because m q^alpha=1. For expert e, X_e is the stochastic number of routed tokens in a window and G_hat_e(z) is its empirical generating function. The proposed loss penalizes disagreement between observed aggregated transforms and this renormalization prediction.

Implementation notes

Integrate the method into the auxiliary loss for a sparse MoE router, before capacity clipping and preferably before deterministic top-k balancing. For each short token window b, sample the router's stochastic assignments and record X[e,b], the number of requests sent to expert e. Use several independent routing replicas if the production router is deterministic. Choose m from {2,4,8} and a transform grid z from {0.2,0.4,0.6,0.8,0.95}. Pseudocode is: compute g1[e,z]=mean_b(zX[e,b]); split or aggregate windows at scale m and compute gm[e,z]; set q=m(-1/alpha); interpolate g1 at zprime=1-q+q*z; compute pred[e,z]=(g1[e,zprime])**m; add lambda_rg*mean((gm-pred)**2) to the task loss. Estimate c and alpha from mean(log(g1[e,z])) by nonlinear least squares, or fix alpha and learn one c per expert. The paper supplies the generating-function transformation and exact fixed-point relation; finite-window bias, dependence between router assignments, and the best alpha must be measured empirically. The first experiment should use a four-expert Switch-Transformer-style MLP on CIFAR-10 or a small language corpus, comparing ordinary load balancing against the proposed loss at batch sizes 256, 512, and 1024. Measure overflow rate, expert-count coefficient of variation, validation loss, and RG residual for m=2 and m=4. The quantitative prediction is that the residual should be approximately invariant across aggregation scales and that normalized expert-load distributions should change by less than 20% when batch size is multiplied by four. A sweep over alpha should show a minimum near the empirically fitted aggregation exponent.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.