Unverified 2026

Hardy–Szegő Repulsive Token Router

Implementation & benchmark of arXiv:2608.17509 — Hardy-Szegő Point Processes: Large Deviations and Strong Szegő Asymptotics

Usefulness6/10

Difficulty6/10

Novelty5/10

Architecture Accuracy Scalability Generalization Moe routing Attention Retrieval Regularization Probability Linear algebra Spectral theory

Source paper: Hardy-Szegő Point Processes: Large Deviations and Strong Szegő Asymptotics arXiv:2608.17509 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace independent top-k token selection by a quality-weighted determinantal subset objective based on the Hardy–Szegő kernel. Tokens with high learned quality are preferred, but geometrically redundant tokens have a small determinant contribution, encouraging diverse sets of routed experts, retrieved items, or attended context tokens.

Formulas

$$K_{\mathbb H}(z,w)=-\frac{1}{\pi(z-\overline{w})^{2}},\qquad z,w\in\mathbb H=\{z\in\mathbb C:\operatorname{Im}z>0\}.$$

$$f_{\mathbb H}(z)=(\varphi^{\prime}(z))^{1/2}\sum_{m=0}^{\infty}\xi_m\varphi(z)^m,\qquad \varphi(z)=\frac{z-i}{z+i}.$$

$$R_{ij}=\frac{K_{\mathbb H}(z_i,z_j)}{\sqrt{K_{\mathbb H}(z_i,z_i)K_{\mathbb H}(z_j,z_j)}}=-\frac{4y_i y_j}{(z_i-\overline{z_j})^2},\qquad z_i=x_i+i y_i,\ y_i>0.$$

$$\det(L_S)=\exp\left(\sum_{i\in S}q_i\right)\det(R_S),\qquad L=DRD,\quad D_{ii}=e^{q_i/2}.$$

Mathematical statement

The paper defines a determinantal process on the upper half-plane $\mathbb H=\{z\in\mathbb C:\operatorname{Im}z>0\}$ with Hermitian kernel $K_{\mathbb H}(z,w)=-1/[\pi(z-\overline w)^2]$. For tokens indexed by $i=1,\ldots,n$, assign coordinates $z_i=x_i+i y_i$ with $y_i>0$, and form the Gram matrix $K_{ij}=K_{\mathbb H}(z_i,z_j)$. Since $K_{ii}=1/(4\pi y_i^2)$, normalize it to $R_{ij}=K_{ij}/\sqrt{K_{ii}K_{jj}}=-4y_i y_j/(z_i-\overline{z_j})^2$, so $R_{ii}=1$. Define a quality-weighted matrix $L=DRD$, where $D_{ii}=\exp(q_i/2)$ and $q_i$ is a router log-quality. A determinantal subset has weight proportional to $\det(L_S)=\exp(\sum_{i\in S}q_i)\det(R_S)$. The quality term favors useful tokens, while the principal-minor determinant penalizes linear dependence and pairwise or higher-order redundancy. Greedy selection uses the Schur complement: after selecting $S$, the marginal determinant contribution of candidate $j$ is $L_{jj}-L_{jS}(L_S+\epsilon I)^{-1}L_{Sj}$.

Implementation notes

1. Integration point: replace the top-k router in a sparse MoE layer, retrieval block, or attention context selector. Given candidate token representations $h_i$, compute a scalar quality logit $q_i=g(h_i)$, and compute coordinate outputs $x_i=a(h_i)$ and $v_i=b(h_i)$. Set $y_i=\operatorname{softplus}(v_i)+y_{\min}$, with $y_{\min}=10^{-3}$, and define $z_i=x_i+i y_i$. For the first experiment, use fixed coordinates derived from normalized token positions so that only the repulsive selection mechanism changes. 2. Pseudocode: q,x,v = router_mlp(h); y = softplus(v)+1e-3; z=x+1j*y; K[i,j] = -1/(pi*(z[i]-conj(z[j]))**2); d=sqrt(real(diag(K))); R=K/(d[:,None]*d[None,:]+eps); R=(R+R.conj().T)/2; L=exp(q[:,None]/2)*R*exp(q[None,:]/2). Initialize S=[]. For each of k selections, evaluate for every unselected candidate j the gain gain[j]=log(max(L[j,j]-L[j,S] @ solve(L[S,S]+eps*I,L[S,j]),eps)), append the largest-gain candidate, and route only the resulting set. 3. The Hardy–Szegő kernel, normalized correlation matrix, determinant objective, and Schur-complement marginal are mathematical components transferred from the paper. The quality logits $q_i$, coordinate network, diagonal jitter $\epsilon$, and greedy approximation are engineering additions. For candidate counts above a few hundred, approximate $R$ with a rank-r Nyström factorization using r=32 or 64 landmarks, then perform greedy selection in the reduced feature space. 4. First cheap experiment: use a small 12-layer Transformer on WikiText-2 or CIFAR-10, replacing a top-8-of-64 context selector or sparse MoE router with this method at identical average selected-token count. Compare against independent top-k and cosine-diversity top-k. Record validation perplexity or accuracy, duplicate-neighbor rate among selected tokens, expert load balance, selection latency, and total training throughput. The method succeeds if it reduces selected-token redundancy and improves validation performance at fixed FLOPs, with no more than 20% router latency overhead. After the fixed-coordinate test, train $x_i$ and $y_i$ jointly and test whether learned upper-half-plane geometry produces an additional gain.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.