Unverified 2026

Calibration-validated uncertainty-aware sparse routing

Implementation & benchmark of arXiv:2608.20295 — Physical-Support Confidence Sets for Highly Coherent Dictionaries

Usefulness6/10

Difficulty6/10

Novelty7/10

Architecture Accuracy Stability Inference speedup Moe routing Retrieval Inference speedup Optimization Statistics Information theory Combinatorics

Source paper: Physical-Support Confidence Sets for Highly Coherent Dictionaries arXiv:2608.20295 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a single plug-in top-k router decision with a confidence correspondence containing every router parameter candidate and sparse expert assignment that remains compatible with calibration and the current input. Project this set onto a hierarchy of expert groups and return the finest group-level decision supported by all surviving explanations; otherwise coarsen the route or abstain. Active endpoint bracketing evaluates only candidates that could still change the projected routing report.

Formulas

$$ (\widehat{D},\widehat{C})\in\arg\min_{D,C}\|Y-DC\|_{F}\quad\text{s.t.}\quad\|c_i\|_{0}\leq k,\ \|d_j\|_{2}=1 $$

$$ \widehat{x}\in\arg\min_x\|Z-\widehat{D}x\|_{2}\quad\text{s.t.}\quad\|x\|_{0}\leq k $$

$$ \delta_{\mathrm{opt}}(N,s)\asymp\min\left\{s,\frac{1}{\sqrt{N}s^{2}}\right\},\qquad \text{orientation information}=Ns^{6} $$

$$ \mathcal C_\varepsilon(Y,Z)=\left\{(D,x):\exists C,\ \|Y-DC\|_F\le\varepsilon_Y,\ \|Z-Dx\|_2\le\varepsilon_Z,\ \|c_i\|_0\le k,\ \|x\|_0\le k,\ \|d_j\|_2=1\right\},\quad \mathcal R=\{P(D,x):(D,x)\in\mathcal C_\varepsilon(Y,Z)\} $$

Mathematical statement

The paper's sparse calibration pipeline is $(\widehat D,\widehat C)\in\arg\min_{D,C}\|Y-DC\|_F$ subject to $\|c_i\|_0\le k$ and $\|d_j\|_2=1$, followed by $\widehat x\in\arg\min_x\|Z-\widehat D x\|_2$ subject to $\|x\|_0\le k$. Here $Y$ is the calibration matrix, $D$ the dictionary, $C$ calibration coefficients, $Z$ a deployment signal, and $x$ its sparse coefficient vector; $k$ is the sparsity budget. The paper argues that coordinate support under one $\widehat D$ need not equal physical support, so we form an adapted finite confidence correspondence $\mathcal C_\varepsilon(Y,Z)=\{(D,x):\|Y-DC\|_F\le\varepsilon_Y,\ \|Z-Dx\|_2\le\varepsilon_Z,\ \|c_i\|_0\le k,\ \|x\|_0\le k,\ \|d_j\|_2=1\}$, where $\varepsilon_Y$ and $\varepsilon_Z$ are calibration and deployment tolerances. For each surviving pair, $P(D,x)$ maps selected expert coordinates to a coarser physical or semantic expert group; the reported set is $\mathcal R=\{P(D,x):(D,x)\in\mathcal C_\varepsilon(Y,Z)\}$. A fine report is valid only when all unresolved candidates induce the same group. The paper's resolution law is $\delta_{\mathrm{opt}}(N,s)\asymp\min\{s,1/(\sqrt N s^2)\}$, where $N$ is the number of calibration signals and $s$ is the separation within a coherent atom block; equivalently the orientation-information scale is $Ns^6$. In routing, this predicts that more calibration helps only when coherent experts differ in a resolvable orientation or feature direction.

Implementation notes

(1) Integration point: modify a top-k MoE router immediately after its logits $r(x)\in\mathbb R^E$ and before dispatching tokens to experts. Partition the $E$ experts into semantic or architectural groups, and maintain a finite bank of router candidates $\{\theta_b\}_{b=1}^B$, obtained by bootstrap retraining, low-rank perturbations, or checkpoints from calibration training. Each candidate gives logits and a k-sparse assignment for the token. (2) Pseudocode: for each calibration minibatch, fit or retain candidate $\theta_b$ only if $L_{cal}(\theta_b)\le L_{cal}^{min}+\varepsilon_Y$. For a deployment token $x$, compute candidate assignments $a_b=TopK(r_{\theta_b}(x),k)$, retain candidates satisfying $L_{dep}(x;\theta_b,a_b)\le L_{dep}^{min}(x)+\varepsilon_Z$, and map each assignment to a group $g_b=P(a_b)$. Start with a fine expert-level report; if the surviving $g_b$ disagree, replace it by their lowest common ancestor in the expert-group hierarchy. If even the root is unsupported by a minimum-confidence threshold, abstain and use a dense fallback or the union of candidate experts. Active endpoint bracketing sorts candidates by the scalar route statistic being evaluated and probes only the current interval endpoints; terminate when all unevaluated candidates are provably unable to alter the group union or common ancestor. (3) Compute the compatibility residuals empirically from held-out calibration loss and per-token reconstruction or task loss; estimate candidate disagreement by bootstrap frequency. The resolution formula is used as a diagnostic: increase the candidate bank or calibration set only when the coherent-group separation estimate $s$ gives $Ns^6$ below one. (4) First experiment: use Switch-Transformer or a small 8-expert top-2 MoE on CIFAR-100 or a 100M-parameter language model, with experts grouped by layer or learned activation similarity. Compare ordinary plug-in top-k routing, bootstrap ensemble routing, and this coarsened-confidence router at equal average expert FLOPs. Measure validation loss, worst-group error, route-flip frequency across checkpoints, number of expert evaluations, and abstention rate. Success means lower accuracy degradation from router perturbations and fewer expert evaluations than evaluating every candidate, while preserving perplexity or task accuracy at matched compute.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.