Unverified 2026

Vacancy-preserving collision-free router

Implementation & benchmark of arXiv:2608.06679 — Tilings, packings, and the existence of Schwartz-class Gabor windows

Usefulness5/10
Difficulty6/10
Novelty5/10

Source paper: Tilings, packings, and the existence of Schwartz-class Gabor windows arXiv:2608.06679 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Build a differentiable assignment layer whose rows represent tokens and whose columns represent experts, memory slots, or attention slots. Each row has unit probability mass, but no column receives positive mass from two rows; maintaining at least one vacant column makes assignments continuously deformable through elementary vacancy moves instead of abrupt softmax switches.

Formulas

$$\lambda_i(a)\geq0,\qquad \#\operatorname{supp}\lambda_i<\infty,\qquad \sum_{a\in A_i}\lambda_i(a)=1$$
$$\lambda_i(a)\lambda_{i'}(b)=0\quad\text{if }a\in A_{i,j},\ b\in A_{i',j}\text{ are distinct labeled addresses}$$
$$C_S=\prod_{i=0}^{m-1}\Delta(S_i),\qquad \Delta(S_i)=\left\{q\geq0:\sum_{a\in S_i}q(a)=1\right\}$$
$$f\left(\sum_k t_kv_k\right)=\sum_k t_kf(v_k),\qquad t_k\geq0,\quad\sum_kt_k=1$$

Mathematical statement

For m labeled rows and n labeled columns, let A_{i,j} be the set of possible addresses assigning row i to column j, and let A_i be the disjoint union of its column-specific address sets. A soft configuration is lambda=(lambda_1,...,lambda_m), where each lambda_i is finitely supported, nonnegative, and sums to one. Collision exclusion requires lambda_i(a)lambda_{i'}(b)=0 whenever distinct addresses a in A_{i,j} and b in A_{i',j} use the same column. For an admissible support pattern S=(S_0,...,S_{m-1}), with each S_i nonempty and no two supported addresses sharing a column, the feasible cell is C_S=product_i Delta(S_i), where Delta(S_i) is the probability simplex on S_i. Affine interpolation inside one common cell preserves feasibility: f(sum_k t_kv_k)=sum_k t_kf(v_k), with t_k nonnegative and summing to one. A hard configuration chooses one address per row. If n>=m+1, at least one column is vacant, so a row can move from its current column to the vacancy, making the old column vacant; repeated vacancy moves connect hard configurations without ever creating a collision.

Implementation notes

Integrate this as a token-to-slot router before a sparse MoE block, slot-attention block, external memory read, or one-to-one cross-attention head. Use a group of m tokens and n>=m+1 slots, with logits z_{i,j} produced by a learned router. Maintain a hard assignment and one vacant slot. At each routing update, greedily choose a target assignment by sorting logits while respecting unused columns. Construct a path from the current assignment to the target: whenever row i must move to a currently occupied target column, first move the row occupying that column into the vacancy, then move row i into the newly vacant column. During a move from column j to vacant column k, use lambda_i=(1-alpha)e_j+alpha e_k for alpha increasing from zero to one; all other rows remain one-hot. Evaluate the sparse module using these weights, or use the interpolation as a short continuation over several training steps. For soft supports, store each row's weights as barycentric coordinates in Delta(S_i), and only interpolate states whose combined support has no repeated column. The exact math is row normalization, column exclusion, and affine feasibility; target selection and move schedules are empirical algorithms. First test m=8,n=9 on a synthetic permutation task, then compare against softmax top-1 routing and Sinkhorn routing in a small MoE classifier or set transformer. Record collision rate, dropped-token rate, routing churn, gradient variance, throughput, and task loss. Success requires zero collisions, smoother loss under token perturbations, and equal accuracy with less overflow or routing overhead than the baseline.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.