✗ Mechanism failed 2026

Allen-Cahn categorical router

Implementation & benchmark of arXiv:2608.26842 — Convergence of a vector-valued Allen-Cahn system to Brakke's multiphase mean curvature flow

Usefulness6/10

Difficulty4/10

Novelty6/10

Regularization Accuracy Stability Generalization Moe routing Quantization Regularization Training dynamics Pde Optimization Dynamical systems Geometry

Source paper: Convergence of a vector-valued Allen-Cahn system to Brakke's multiphase mean curvature flow arXiv:2608.26842 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the usual softmax router or soft one-hot penalty with a vector-valued phase-field regularizer whose low-energy states are exactly the expert one-hot vectors. Component-wise barriers create stable categorical phases, while a weaker coupling term suppresses invalid states such as the all-zero vector or multi-expert activation; annealing $\varepsilon$ produces increasingly discrete routing.

Formulas

$$W(u)=u^{2}(1-u)^{2},\qquad \phi(u)=\int_{0}^{u}\sqrt{2W(s)}\,\mathrm{d}s,\qquad \sigma=\phi(1).$$

$$F_{\varepsilon}(\mathbf{u})=\frac{1}{\varepsilon}\sum_{i=1}^{n}W(u_i)+\frac{1}{2\varepsilon^{\alpha}}\left(\sigma-\sum_{i=1}^{n}\phi(u_i)\right)^{2},\qquad \alpha\in(0,1).$$

$$w_i=\frac{\phi(u_i)}{\sigma},\qquad \frac{\sigma^{2}}{2\varepsilon^{\alpha}}\left(1-\sum_{i=1}^{n}w_i\right)^{2}.$$

$$\mathcal L(\theta)=\mathcal L_{\mathrm{task}}(\theta)+\lambda F_{\varepsilon}(\mathbf u_\theta),\qquad \mathbf u_\theta=\operatorname{sigmoid}(\mathbf a_\theta),\qquad \varepsilon_{k+1}=\max(\varepsilon_{\min},\rho\varepsilon_k),\quad 0<\rho<1.$$

Mathematical statement

The paper uses the scalar double-well potential $W(u)=u^{2}(1-u)^{2}$, its phase coordinate $\phi(u)=\int_{0}^{u}\sqrt{2W(s)}\,\mathrm{d}s$, and the normalization $\sigma=\phi(1)$. For a router output $\mathbf u=(u_{1},\ldots,u_{n})\in[0,1]^{n}$, the multi-well potential is $F_{\varepsilon}(\mathbf u)=\frac{1}{\varepsilon}\sum_{i=1}^{n}W(u_i)+\frac{1}{2\varepsilon^{\alpha}}\left(\sigma-\sum_{i=1}^{n}\phi(u_i)\right)^{2}$, with $\alpha\in(0,1)$. The first term independently drives every component toward 0 or 1; the second term weakly enforces exactly one active component in transformed coordinates $w_i=\phi(u_i)/\sigma$, because it becomes $\frac{\sigma^{2}}{2\varepsilon^{\alpha}}(1-\sum_i w_i)^2$. The intended minima are the standard basis vectors $\mathbf e_i$, where one component equals 1 and all others equal 0. In the neural adaptation, $\mathbf u$ is a sigmoid router output, $\alpha$ controls scale separation, $\varepsilon$ is annealed, and the energy is added to the task loss.

Implementation notes

Integrate the method at the router-logit tensor before expert dispatch. For each token representation x, compute a=RouterMLP(x) in R^n and u=sigmoid(a), rather than immediately using softmax probabilities. For this W on [0,1], evaluate phi analytically as phi(u)=sqrt(2)*(u^2/2-u^3/3), and set sigma=phi(1). Use the following loop: a=router(x); u=sigmoid(a); W=u**2*(1-u)**2; phi=sqrt(2)*(u2/2-u3/3); F=W.sum(dim=-1)/eps + (sigma-phi.sum(dim=-1))**2/(2*eps**alpha); loss=task_loss + lambda*F.mean(); backprop(loss); optimizer.step(); eps=max(eps_min,rho*eps). During training, use soft weighted expert outputs or top-1 dispatch with a straight-through estimator; at evaluation, route by argmax(u). No paper-specific spectral or geometric quantity needs estimation. Start with alpha=0.5 and epsilon=1, then geometrically anneal to 0.02 over the first 30-50% of training. Tune lambda so the regularizer is initially 1-10% of task loss. Record routing entropy, the fraction of tokens with max(u)>0.95, expert load imbalance, number of active experts, validation loss, and gradient norms. First test a small Switch-Transformer-style MoE on CIFAR-100 or a toy language-model corpus, comparing against softmax top-1 routing plus standard load balancing at equal parameters and FLOPs. Success is lower routing entropy and fewer multi-expert assignments without expert collapse, while validation loss matches or improves over baseline; a secondary success criterion is stable optimization during late-stage epsilon annealing.

Verification

Mechanism failed

Stage 1 · Toy mechanism gate: Failed ✗

Stage 2 · Mechanism transferred to benchmark: Not tested

Stage 2 · Practical benchmark result: Not run

Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built and tested an Allen–Cahn sigmoid router with analytic phase coordinate, epsilon annealing, synthetic MoE training, and routing diagnostics. The formula check passed with maximum derivative error 1.97e-7, and the idea produced lower entropy, fully high-confidence routing, and slightly better validation loss than baseline. However, it caused severe expert imbalance (load CV 1.33 vs 0.91), used only 2 of 4 experts, and unconstrained relaxation often converged to multi-active states for n=4, so the categorical-routing promise was not achieved robustly.

Agent confidence: 8/10

Baseline: Validation loss 0.01413; accuracy 0.9940; routing entropy 0.5039; fraction max(u)>0.95 = 0.372; load imbalance CV 0.907; active experts 3.
Idea: Validation loss 0.01283; accuracy 0.9960; routing entropy 0.4027; fraction max(u)>0.95 = 1.000; load imbalance CV 1.327; active experts 2; final epsilon 0.02.
Math check: Analytic phi derivative maximum error 1.97e-7. Direct one-hot energy was approximately 3.2e-16, versus zero-vector energy 0.0802 and n=4 all-one energy 0.7217.

Limitations: Only a small synthetic binary-classification MoE was tested on CPU. CIFAR-100, language modeling, true top-1 dispatch, straight-through estimation, FLOP-normalized comparison, multiple seeds, and systematic lambda or annealing tuning were not evaluated.

How to run: python3 experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Artifacts

Implementation overview ⬇ Download all as ZIP 2 files · code, reports and structured results

🐍 experiment.py 4.1 KB View Raw
📄 report.md 1.5 KB View