# Allen-Cahn categorical router

- ID: 212
- Canonical URL: https://synthcore.org/idea/212/allen-cahn-categorical-router
- API JSON: https://synthcore.org/api/idea/212.json
- API Markdown: https://synthcore.org/api/idea/212.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.26842](https://arxiv.org/abs/2608.26842)
- Category: regularization
- Solves: accuracy, stability, generalization
- ML areas: moe-routing, quantization, regularization, training-dynamics
- Math tags: pde, optimization, dynamical-systems, geometry
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Replace the usual softmax router or soft one-hot penalty with a vector-valued phase-field regularizer whose low-energy states are exactly the expert one-hot vectors. Component-wise barriers create stable categorical phases, while a weaker coupling term suppresses invalid states such as the all-zero vector or multi-expert activation; annealing \(\varepsilon\) produces increasingly discrete routing.

## Mathematical statement

The paper uses the scalar double-well potential \(W(u)=u^{2}(1-u)^{2}\), its phase coordinate \(\phi(u)=\int_{0}^{u}\sqrt{2W(s)}\,\mathrm{d}s\), and the normalization \(\sigma=\phi(1)\). For a router output \(\mathbf u=(u_{1},\ldots,u_{n})\in[0,1]^{n}\), the multi-well potential is \(F_{\varepsilon}(\mathbf u)=\frac{1}{\varepsilon}\sum_{i=1}^{n}W(u_i)+\frac{1}{2\varepsilon^{\alpha}}\left(\sigma-\sum_{i=1}^{n}\phi(u_i)\right)^{2}\), with \(\alpha\in(0,1)\). The first term independently drives every component toward 0 or 1; the second term weakly enforces exactly one active component in transformed coordinates \(w_i=\phi(u_i)/\sigma\), because it becomes \(\frac{\sigma^{2}}{2\varepsilon^{\alpha}}(1-\sum_i w_i)^2\). The intended minima are the standard basis vectors \(\mathbf e_i\), where one component equals 1 and all others equal 0. In the neural adaptation, \(\mathbf u\) is a sigmoid router output, \(\alpha\) controls scale separation, \(\varepsilon\) is annealed, and the energy is added to the task loss.

## Key formulas

- $$W(u)=u^{2}(1-u)^{2},\qquad \phi(u)=\int_{0}^{u}\sqrt{2W(s)}\,\mathrm{d}s,\qquad \sigma=\phi(1).$$
- $$F_{\varepsilon}(\mathbf{u})=\frac{1}{\varepsilon}\sum_{i=1}^{n}W(u_i)+\frac{1}{2\varepsilon^{\alpha}}\left(\sigma-\sum_{i=1}^{n}\phi(u_i)\right)^{2},\qquad \alpha\in(0,1).$$
- $$w_i=\frac{\phi(u_i)}{\sigma},\qquad \frac{\sigma^{2}}{2\varepsilon^{\alpha}}\left(1-\sum_{i=1}^{n}w_i\right)^{2}.$$
- $$\mathcal L(\theta)=\mathcal L_{\mathrm{task}}(\theta)+\lambda F_{\varepsilon}(\mathbf u_\theta),\qquad \mathbf u_\theta=\operatorname{sigmoid}(\mathbf a_\theta),\qquad \varepsilon_{k+1}=\max(\varepsilon_{\min},\rho\varepsilon_k),\quad 0<\rho<1.$$

## Implementation notes

Integrate the method at the router-logit tensor before expert dispatch. For each token representation x, compute a=RouterMLP(x) in R^n and u=sigmoid(a), rather than immediately using softmax probabilities. For this W on [0,1], evaluate phi analytically as phi(u)=sqrt(2)*(u^2/2-u^3/3), and set sigma=phi(1). Use the following loop: `a=router(x); u=sigmoid(a); W=u**2*(1-u)**2; phi=sqrt(2)*(u**2/2-u**3/3); F=W.sum(dim=-1)/eps + (sigma-phi.sum(dim=-1))**2/(2*eps**alpha); loss=task_loss + lambda*F.mean(); backprop(loss); optimizer.step(); eps=max(eps_min,rho*eps)`. During training, use soft weighted expert outputs or top-1 dispatch with a straight-through estimator; at evaluation, route by argmax(u). No paper-specific spectral or geometric quantity needs estimation. Start with alpha=0.5 and epsilon=1, then geometrically anneal to 0.02 over the first 30-50% of training. Tune lambda so the regularizer is initially 1-10% of task loss. Record routing entropy, the fraction of tokens with max(u)>0.95, expert load imbalance, number of active experts, validation loss, and gradient norms. First test a small Switch-Transformer-style MoE on CIFAR-100 or a toy language-model corpus, comparing against softmax top-1 routing plus standard load balancing at equal parameters and FLOPs. Success is lower routing entropy and fewer multi-expert assignments without expert collapse, while validation loss matches or improves over baseline; a secondary success criterion is stable optimization during late-stage epsilon annealing.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built and tested an Allen–Cahn sigmoid router with analytic phase coordinate, epsilon annealing, synthetic MoE training, and routing diagnostics. The formula check passed with maximum derivative error 1.97e-7, and the idea produced lower entropy, fully high-confidence routing, and slightly better validation loss than baseline. However, it caused severe expert imbalance (load CV 1.33 vs 0.91), used only 2 of 4 experts, and unconstrained relaxation often converged to multi-active states for n=4, so the categorical-routing promise was not achieved robustly.

### Mechanism check

- Verdict: Built and tested an Allen–Cahn sigmoid router with analytic phase coordinate, epsilon annealing, synthetic MoE training, and routing diagnostics. The formula check passed with maximum derivative error 1.97e-7, and the idea produced lower entropy, fully high-confidence routing, and slightly better validation loss than baseline. However, it caused severe expert imbalance (load CV 1.33 vs 0.91), used only 2 of 4 experts, and unconstrained relaxation often converged to multi-active states for n=4, so the categorical-routing promise was not achieved robustly.
- Confidence: 8/10
- Limitations: Only a small synthetic binary-classification MoE was tested on CPU. CIFAR-100, language modeling, true top-1 dispatch, straight-through estimation, FLOP-normalized comparison, multiple seeds, and systematic lambda or annealing tuning were not evaluated.

## Artifacts

- [experiment.py](https://synthcore.org/code/34/experiment.py)
- [report.md](https://synthcore.org/code/34/report.md)
- [Download all files as ZIP](https://synthcore.org/download/34)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.