# Lorentzian interaction router

- ID: 3089
- Canonical URL: https://synthcore.org/idea/3089/lorentzian-interaction-router
- API JSON: https://synthcore.org/api/idea/3089.json
- API Markdown: https://synthcore.org/api/idea/3089.md
- Verification status: unverified
- Source: [arXiv:2609.02850](https://arxiv.org/abs/2609.02850)
- Category: regularization
- Solves: stability, generalization, accuracy
- ML areas: moe-routing, attention, regularization
- Math tags: convex-analysis, linear-algebra, geometry
- Ratings: usefulness 6/10; difficulty 6/10; novelty 8/10

## Idea description

Replace an unconstrained positive higher-order gating function by a factorially normalized homogeneous polynomial whose coefficients are trained toward the Lorentzian Hessian signature. This gives multiplicative routing or attention interactions a controlled curvature pattern instead of allowing arbitrary unstable higher-order amplification.

## Mathematical statement

The paper defines a homogeneous polynomial f(x)=sum_{|alpha|=d} c_alpha x^alpha with c_alpha>=0, the divided-power monomial x^[alpha]=x^alpha/alpha!, and the normalized polynomial N(f)=sum_{|alpha|=d} c_alpha x^[alpha]. Here alpha=(alpha_1,...,alpha_n), |alpha|=sum_i alpha_i=d, and alpha!=product_i alpha_i!. The Lorentzian property means that after applying any d-2 directional derivatives in nonnegative directions, the resulting degree-2 polynomial has a Hessian with at most one positive eigenvalue. For sampled nonnegative directions u_1,...,u_{d-2}, define H_u(x)=nabla_x^2[(u_1 dot nabla)...(u_{d-2} dot nabla)N(f)(x)]. The neural regularizer penalizes the second-largest eigenvalue lambda_2(H_u), so the target condition is lambda_2(H_u)<=0. The factorial normalization matters because it makes coefficients correspond directly to mixed intersection-volume coefficients and avoids degree-dependent multiplicity factors.

## Key formulas

- $$f=\sum_{|\alpha|=d}c_{\alpha}x^{\alpha},\qquad x^{[\alpha]}:=\frac{x^{\alpha}}{\alpha!},\qquad\mathrm{N}(f):=\sum_{|\alpha|=d}c_{\alpha}x^{[\alpha]},\qquad\alpha!:=\prod_{i}\alpha_{i}!.$$
- $$H_{u_1,\ldots,u_{d-2}}(x):=\nabla_x^2\left[(u_1\cdot\nabla_x)\cdots(u_{d-2}\cdot\nabla_x)\mathrm{N}(f)(x)\right],\qquad \lambda_2\left(H_{u_1,\ldots,u_{d-2}}(x)\right)\le 0.$$
- $$\mathcal{L}_{\mathrm{Lor}}=\frac{1}{B}\sum_{b=1}^{B}\max\left(0,\lambda_2(H_b)\right)^2.$$
- $$c_{\alpha}=\operatorname{softplus}(\theta_{\alpha}),\qquad g(z)=\log\left(\varepsilon+\mathrm{N}(f)(\operatorname{softplus}(z))\right).$$

## Implementation notes

Integrate the module into a higher-order MoE router or low-rank attention bias. Let each token representation h produce z=softplus(Wh+b) in R^n. Store trainable logits theta_alpha for all multi-indices alpha with |alpha|=d, set c_alpha=softplus(theta_alpha), and compute the interaction score g(z)=log(epsilon+sum_alpha c_alpha z^alpha/alpha!). Add g(z) either to every expert logit through an expert-specific coefficient tensor or use it as a shared load-aware router bias. During each training step, sample B feature points z_b from the current minibatch and nonnegative directions u_{b,1},...,u_{b,d-2}; use normalized softplus random projections or fixed coordinate mixtures. Compute the directional derivative polynomial with automatic differentiation, form its n-by-n Hessian H_b, and add lambda_L times max(0,lambda_2(H_b))^2 to the task loss. For d=3, only one directional derivative is needed; for d=4, use two. The paper's mathematics supplies the factorial representation and the one-positive-eigenvalue criterion. Estimate lambda_2 with a symmetric eigensolver for n<=32, or approximate it using a differentiable eigensolver/power iteration. Test n=16,d=3 in a small Switch-style MoE on CIFAR-100 or WikiText-103, comparing a standard softmax router, an unconstrained positive polynomial router, and the Lorentzian-regularized router at equal FLOPs. Track validation loss, expert-load coefficient of variation, router entropy, gradient norm spikes, and training time. Success is lower load variance and fewer instability spikes with equal or better validation loss; reject it if the curvature penalty costs over 10% training time without improving stability.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.