# Rayleigh–Jeans Condensation Routing

- ID: 549
- Canonical URL: https://synthcore.org/idea/549/rayleigh-jeans-condensation-routing
- API JSON: https://synthcore.org/api/idea/549.json
- API Markdown: https://synthcore.org/api/idea/549.md
- Verification status: unverified
- Source: [arXiv:2607.07315](https://arxiv.org/abs/2607.07315)
- Category: architecture
- Solves: accuracy, scalability
- ML areas: attention, moe-routing, transformer
- Math tags: probability, statistics, optimization
- Ratings: usefulness 5/10; difficulty 6/10; novelty 7/10

## Idea description

Replace softmax attention or dense MoE routing with a normalized Rayleigh–Jeans distribution over tokens or experts. If an item's energy is close to the chemical potential, its probability becomes disproportionately large, creating controllable low-energy condensation instead of the exponentially smooth allocation produced by softmax.

## Mathematical statement

The paper uses the Rayleigh–Jeans law \(\rho_m=T/(E_m-\mu)\), where \(m\) indexes modes, \(E_m\) is the mode energy, \(T\) is temperature, \(\mu\) is chemical potential, and \(\rho_m\) is the allocated mass. The parameters are fixed by conservation of total mass and mean energy: \(\sum_{m=1}^{M}\rho_m=1\) and \(\sum_{m=1}^{M}E_m\rho_m=\bar E\), where \(M\) is the number of candidates and \(\bar E\) is a chosen target mean energy. For the positive-temperature regime, require \(\mu<\min_m E_m\) and \(T>0\), which guarantees positive probabilities. Eliminating \(T\) gives \(T(\mu)=1/\sum_m(E_m-\mu)^{-1}\); therefore, \(\mu\) can be found by solving \(\sum_m E_m/(E_m-\mu)\,/\,\sum_m1/(E_m-\mu)=\bar E\). As \(\mu\) approaches the smallest energy, the lowest-energy candidate receives a sharply increasing share, giving the condensation effect. In the neural adaptation, energies are defined as \(E_j=c-s_j\), with \(s_j\) an attention similarity or negative expert routing score and \(c>\max_j s_j\) a positivity offset.

## Key formulas

- $$\rho_m=\frac{T}{E_m-\mu}\qquad\text{(Rayleigh--Jeans law)}$$
- $$\sum_{m=1}^{M}\rho_m=1,\qquad \sum_{m=1}^{M}E_m\rho_m=\bar E$$
- $$T(\mu)=\left(\sum_{m=1}^{M}\frac{1}{E_m-\mu}\right)^{-1},\qquad \frac{\sum_{m=1}^{M}\frac{E_m}{E_m-\mu}}{\sum_{m=1}^{M}\frac{1}{E_m-\mu}}=\bar E$$
- $$E_j=c-s_j,\qquad p_j=\frac{(E_j-\mu)^{-1}}{\sum_{k=1}^{M}(E_k-\mu)^{-1}}$$

## Implementation notes

Integrate this at the attention-weight or MoE-router probability computation, initially for a small experimental branch rather than replacing every softmax. For attention, compute similarities \(s_{ij}=q_i^\top k_j/\sqrt d\) for query \(i\), set \(E_{ij}=c_i-s_{ij}\), and choose \(c_i=\max_j s_{ij}+\epsilon\) with \(\epsilon>0\). Choose a target mean energy \(\bar E_i\), initially by matching the mean energy of the baseline softmax attention at temperature 1, or use a learned scalar constrained to lie between the minimum and maximum candidate energies. Solve for \(\mu_i<\min_jE_{ij}\) by bisection on the scalar equation \(g(\mu)=\sum_jE_{ij}/(E_{ij}-\mu)\,/\,\sum_j1/(E_{ij}-\mu)-\bar E_i=0\). Then compute \(T_i=1/\sum_j1/(E_{ij}-\mu_i)\) and probabilities \(p_{ij}=T_i/(E_{ij}-\mu_i)\). In practice use \(\mu_i=\min_jE_{ij}-\delta\), with \(\delta\) bounded below, if the root is numerically close to the pole. For MoE, replace \(j\) by experts, use router scores as \(s_j\), and select top-k experts from \(p_j\), optionally renormalizing the selected probabilities. The mathematical quantities are the inverse-gap probabilities, normalization, mean-energy root, and pole-distance constraint; the target \(\bar E\), offset \(c\), and minimum gap \(\delta\) are engineering hyperparameters or learned values. The first cheap experiment should compare softmax, sparsemax, and this Rayleigh–Jeans router in a 6-layer Transformer on a small language-model corpus, measuring validation loss, attention entropy, effective support, and wall-clock cost. A successful signal is lower effective support at equal validation loss, or lower validation loss at equal FLOPs, without unstable gradients when the best score changes between tokens.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.