# Rayleigh-Jeans Condensing Router

- ID: 950
- Canonical URL: https://synthcore.org/idea/950/rayleigh-jeans-condensing-router
- API JSON: https://synthcore.org/api/idea/950.json
- API Markdown: https://synthcore.org/api/idea/950.md
- Verification status: unverified
- Source: [arXiv:2607.15119](https://arxiv.org/abs/2607.15119)
- Category: architecture
- Solves: scalability, sample-efficiency, accuracy
- ML areas: moe-routing, training-dynamics, scheduler
- Math tags: statistical-mechanics, optimization, probability, dynamical-systems
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace a standard softmax MoE router with a thermodynamic router whose expert occupations maximize entropy subject to a prescribed total routing mass and mean routing energy. At high temperature, traffic is distributed across many experts; as temperature decreases or the energy budget tightens, traffic undergoes a predictable condensation transition in which excess load moves to the lowest-energy expert or expert group. This supplies an explicit control knob for adaptive specialization instead of relying only on unconstrained softmax competition.

## Mathematical statement

Let K experts have nonnegative occupations n_i, mode energies epsilon_i, total occupation N = sum_i n_i, and energy E = sum_i epsilon_i n_i. Maximizing the classical entropy S = sum_i[(1+n_i) log(1+n_i) - n_i log(n_i)] subject to fixed N and E gives the Rayleigh-Jeans occupation n_i = T/(epsilon_i - mu), where T = 1/beta is temperature and mu is a chemical potential satisfying mu < epsilon_min. If mu reaches the minimum energy epsilon_0, the excited-state capacity is N_ex(T) = sum over i with epsilon_i greater than epsilon_0 of T/(epsilon_i - epsilon_0). Whenever N > N_ex(T), the residual n_0 = N - N_ex(T) is a condensate in the minimum-energy mode. In an MoE batch, n_i is aggregate gate probability sent to expert i, while epsilon_i can be measured latency, recent loss, or a learned specialization cost.

## Key formulas

- S(n) = sum_i [(1+n_i) log(1+n_i) - n_i log(n_i)],  N = sum_i n_i,  E = sum_i epsilon_i n_i
- d/dn_i [S - beta E + beta mu N] = 0  =>  n_i_RJ = T/(epsilon_i - mu),  T = 1/beta,  mu < epsilon_min
- N_ex(T) = sum_{i: epsilon_i > epsilon_0} T/(epsilon_i - epsilon_0),  n_0 = max(0, N - N_ex(T)),  epsilon_0 = min_i epsilon_i
- L_route = lambda_RJ D_KL(n_hat/N || n_RJ/N) + lambda_E (sum_i epsilon_i n_hat_i - E_star)^2 + lambda_N (sum_i n_hat_i - N)^2

## Implementation notes

1. Integration point: insert this router immediately before expert dispatch in a sparse MoE layer. For a batch of B tokens, compute ordinary router logits a_bi, convert them to soft assignments r_bi = softmax_i(a_b), and define batch occupations n_hat_i = sum_b r_bi. Define each expert energy as epsilon_i = c_latency times normalized measured latency_i plus c_loss times EMA(expert loss_i) minus c_utility times EMA(expert utility_i); normalize all components across experts. 2. Pseudocode: initialize T and mu below min(epsilon). On each batch, estimate n_hat, solve sum_i T/(epsilon_i - mu) = N and sum_i epsilon_i T/(epsilon_i - mu) = E_star using bisection or Newton iterations, and form n_RJ. Add the KL, energy, and mass penalties to the task loss. For dispatch, use adjusted logits a_prime_bi = a_bi + log(n_RJ_i + small_constant) - log(n_hat_i + small_constant), then apply top-1 or top-2 routing. Anneal T only after experts have learned distinct functions. 3. Computed directly from the mechanism: the Rayleigh-Jeans target, chemical-potential solve, and critical excited capacity. Estimated empirically: expert energies, occupations, and task-loss response. 4. First cheap experiment: train a 4- or 8-expert top-2 MLP-MoE on CIFAR-100 or a small language-model corpus, comparing standard softmax routing, Switch load balancing, and this router. Sweep T and E_star. Fit observed occupations to T/(epsilon_i - mu) and measure condensate fraction. The quantitative prediction is that when N crosses N_ex(T), the minimum-energy expert's load exhibits a sharp increase while the remaining loads follow approximately 1/(epsilon_i - epsilon_0). The measured transition should be within 20 percent of the numerically solved threshold. Also measure whether controlled condensation reduces communication and routing entropy without increasing validation loss.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.