Unverified 2026

Density-annealed Coulomb embedding repulsion

Implementation & benchmark of arXiv:2608.16655 — Quantitative mean-field limits for repulsive Coulomb flows at bounded density and Riesz weak--strong stability

Usefulness6/10

Difficulty4/10

Novelty5/10

Regularization Accuracy Stability Generalization Embedding Moe routing Regularization Dynamical systems Pde Optimal transport Numerical analysis

Source paper: Quantitative mean-field limits for repulsive Coulomb flows at bounded density and Riesz weak--strong stability arXiv:2608.16655 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Treat trainable prototypes, class centers, codebook entries, or router expert embeddings as interacting particles and add a mollified repulsive Coulomb force to their task-gradient update. Unlike a fixed repulsion coefficient, use the paper's explicit density envelope to reduce repulsion over training and use the associated density-dependent mollification radius, so early training prevents collapse while late training permits precise cluster formation.

Formulas

$$\dot{x}_{i}(t)=-\frac{1}{N}\sum_{j\neq i}\nabla g(x_{i}(t)-x_{j}(t)),\qquad 1\leq i\leq N.$$

$$\partial_{t}\rho=\nabla\!\cdot(\rho u),\qquad u=\nabla g*\rho,$$

$$m(t)=\frac{\|\rho_{0}\|_{L^{\infty}}}{1+t\|\rho_{0}\|_{L^{\infty}}},\qquad m'(t)=-m(t)^2.$$

$$z_i^{+}=z_i-\eta\left(\nabla_{z_i}\mathcal{L}_{\mathrm{task}}+\lambda_0\frac{m(t)}{M}\,r_i\right),\quad r_i=\frac{1}{N}\sum_{j\neq i}\nabla g_{\varepsilon_t}(z_i-z_j),\quad \varepsilon_t=\kappa m(t)^{-1/d}.$$

Mathematical statement

The particle dynamics in the paper are $\dot{x}_i=-(1/N)\sum_{j\ne i}\nabla g(x_i-x_j)$, where $x_i\in\mathbb{R}^d$ is particle $i$, $N$ is the particle count, and $g$ is an even repulsive interaction potential. The limiting density satisfies $\partial_t\rho=\nabla\cdot(\rho u)$ with $u=\nabla g*\rho$; characteristics therefore move with velocity $-u$. For the Coulomb flow, the paper gives the bounded-density envelope $m(t)=M/(1+tM)$, $m'(t)=-m(t)^2$, where $M=\|\rho_0\|_{\infty}$ is the initial density bound. The paper also uses mollification at scale $m(t)^{-1/d}$, with $d$ the particle-space dimension, and controls force mismatch through a negative mean-square force-error term. The neural adaptation uses the same particle force with a smooth kernel $g_\varepsilon$, sets $\varepsilon_t=\kappa m(t)^{-1/d}$, and makes the repulsion coefficient proportional to $m(t)/M$. For numerical stability, the singular force is evaluated as $r_i=(1/N)\sum_{j\ne i}\nabla g_{\varepsilon_t}(z_i-z_j)$, where $z_i$ are normalized trainable embeddings.

Implementation notes

Integrate this at the update of a set of trainable vectors: class prototypes in prototypical networks, learnable token embeddings, vector-quantization codebook entries, or MoE expert/router embeddings. Do not apply it to every hidden activation initially; begin with one matrix Z of shape N by d that is explicitly optimized. First L2-normalize each row of Z and optionally whiten it over the minibatch so that the density schedule has a meaningful coordinate scale. Set M to an initial density estimate, for example M=N/(V_d R^d), where R is the radius containing 95% of the normalized vectors and V_d is the d-dimensional unit-ball volume. At optimizer step k, compute m=M/(1+kM/S), where S is the intended repulsion-decay horizon, and set epsilon=kappa times m^{-1/d}; clamp epsilon to [epsilon_min, epsilon_max]. Compute pairwise differences D[i,j]=Z[i]-Z[j], evaluate the gradient of a mollified Coulomb kernel, and average over j not equal to i to obtain r_i. For d=2 use g_epsilon(x)=-(1/(2 pi)) log(sqrt(||x||^2+epsilon^2)); for d greater than 2 use g_epsilon(x)=c_d (||x||^2+epsilon^2)^((2-d)/2), with c_d chosen as 1/((d-2)|S^{d-1}|) or treated as a tunable global scale. Add lambda_0(m/M)r_i to the task gradient before AdamW or SGD. For large N, estimate r_i from 4 to 16 random neighbors or a spatial hash rather than an all-pairs matrix. The paper supplies the decay law and scale relation; estimate M, the radius, and force-subsampling error empirically. The first experiment should use a CIFAR-100 prototypical classifier or a 32-expert MoE on a small language-model benchmark, comparing no repulsion, fixed Gaussian repulsion, and this schedule at equal FLOPs. Measure prototype or expert pairwise-distance collapse, task accuracy, gradient-norm variance, and late-training loss. Success is higher validation accuracy or routing utilization with no increase in early instability, plus a lower repulsion force and sharper within-class clusters late in training than the fixed-repulsion baseline.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.