✗ Failed on benchmark 2026

Spectral placement of expensive verifiers

Implementation & benchmark of arXiv:2607.09765 — How Much Does Correctness Cost? Budgeted Placement of Strong Correctors in a Weak Multi-Agent Swarm

Usefulness7/10

Difficulty5/10

Novelty7/10

Architecture Accuracy Scalability Speedup Moe routing Inference speedup Rlhf Graph nn Graph theory Linear algebra Spectral theory Optimization

Source paper: How Much Does Correctness Cost? Budgeted Placement of Strong Correctors in a Weak Multi-Agent Swarm arXiv:2607.09765 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Construct a graph of cheap prediction agents or reasoning traces and use a sparse set of expensive verifier calls as graph anchors. Select the next verifier location by the exact reduction in a trace-inverse coherence objective per unit cost, rather than by uncertainty or random sampling. This creates a budgeted mixture-of-agents architecture that can spend computation where it most improves global consensus.

Formulas

$$M(R)=L+\kappa I+\sum_{i\in R}w_i e_i e_i^{\top}=M_0+W_R,\qquad W_R=\operatorname{diag}\!\big(w_i\,\mathds{1}[i\in R]\big).$$

$$H(R)=\operatorname{tr}M(R)^{-1}.$$

$$H(R)-H(R\cup\{i\})=\frac{w_i\,e_i^{\top}M(R)^{-2}e_i}{1+w_i\,e_i^{\top}M(R)^{-1}e_i}.$$

$$i_t=\arg\max_{i\notin R_t,\ c_i\leq B_t}\frac{H(R_t)-H(R_t\cup\{i\})}{c_i},\qquad R_{t+1}=R_t\cup\{i_t\}.$$

Mathematical statement

The paper defines a positive-definite baseline operator M_0=L+\kappa I, where L is the graph Laplacian over K cheap agents, I is the K-by-K identity, and \kappa>0 is a diagonal damping parameter. Selecting a verifier at candidate node i adds a rank-one pin w_i e_i e_i^ op, where e_i is the i-th standard basis vector and w_i>0 is verifier strength, giving M(R)=M_0+\sum_{i\in R}w_i e_i e_i^ op. The coherence objective is H(R)=\operatorname{tr}(M(R)^{-1}); lower H means less expected consensus error. For a current placement R, let A=M(R)^{-1}. The Sherman-Morrison identity gives the exact marginal decrease from adding candidate i: H(R)-H(R\cup\{i\})=\frac{w_i e_i^ op A^2 e_i}{1+w_i e_i^ op A e_i}. The numerator measures how strongly perturbations at node i propagate through the network, while the denominator accounts for saturation from an already strong pin. With verifier cost c_i, use marginal gain divided by c_i and greedily add feasible candidates; the paper's submodularity result supplies diminishing returns and a 1-1/e-style greedy justification under a cardinality or knapsack budget.

Implementation notes

Integrate this at inference time into a multi-agent or mixture-of-experts predictor. First generate K cheap candidate answers, trajectories, or logits from small models; treat each candidate as one graph node. Build a symmetric similarity matrix S from normalized hidden-state cosine similarity, answer agreement, or pairwise logit correlation, and set L=D-S with D_{ii}=\sum_j S_{ij}. Choose \kappa, verifier strengths w_i, and costs c_i from calibration data. At each budget step, form M=L+\kappa I+\sum_{j\in R}w_j e_j e_j^ op. Do not explicitly invert M for large K: estimate H and each marginal with conjugate-gradient solves. For Hutchinson probes z_r with Rademacher entries, solve Mx_r=z_r and estimate H\approx(1/m)\sum_r z_r^ op x_r. For candidate i, solve M y_i=e_i, or reuse factorizations, and compute e_i^ op M^{-1}e_i=e_i^ op y_i and e_i^ op M^{-2}e_i=\|y_i\|_2^2; then rank candidates by w_i\|y_i\|_2^2/((1+w_i e_i^ op y_i)c_i). The selected verifier runs on that candidate and its corrected logits are injected as a pin; recompute the graph consensus by solving Mx=b independently for each logit dimension, then aggregate x or average its softmax. Estimate L and w_i empirically, but compute the marginal score from the paper's formula. Start with K=16 cheap small-model samples on GSM8K or HumanEval, comparing random, entropy, and confidence-based verifier allocation at 1, 2, 4, and 8 verifier calls. The success signal is higher exact-match or pass@1 at equal verifier FLOPs, or the same accuracy with fewer expensive calls; additionally verify that greedy gains decrease over successive calls as predicted by submodularity.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a NumPy MVP for spectral verifier placement with exact Sherman–Morrison marginals, cost-aware greedy selection, and synthetic baselines. The mechanism manifested: observed marginals matched the formula within 6.6e-16 absolute error, increased monotonically with verifier strength, and reached 99.74% of the predicted saturation ceiling at w=1000; exhaustive checks over all 1024 subsets found no diminishing-returns violation. In the synthetic budget experiment, greedy achieved mean H=3.542 versus 3.829 for random and 3.589 for entropy, but this is a coherence-objective result rather than an end-to-end task-accuracy win.

Agent confidence: 9/10

Baseline: Random mean H=3.8291 (std 0.2138), entropy mean H=3.5891 (std 0.1825), budget 8, 100 trials
Idea: Greedy exact marginal/cost mean H=3.5425 (std 0.1802), mean spent cost 7.77, mean verifier calls 6.51; identity max relative error 4.78e-13; saturation max absolute formula error 6.61e-16; observed final/ceiling=0.9974; minimum later/earlier marginal ratio=0.6345

Limitations: No GSM8K/HumanEval or real multi-agent verifier was tested, and no wall-clock/CUDA performance or actual prediction accuracy was measured. The graph, verifier strengths, costs, and entropy proxy are synthetic; the submodularity check is exhaustive only for K=10 and the allocation benchmark uses coherence H as its outcome metric.

How to run: python3 spectral_verifier.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.09765) Aug 31, 2026

No significant difference vs. the tuned baseline (+2.1%, p=0.6508, wins 4 / 8 paired seeds).

Benchmark: Verifier_portfolio
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 0.1628
Idea mean: 0.1662
Effect (Δ): 0.0034 (+2.1%; negative = idea better)
Wins: 4 / 8 paired seeds
p-value: 0.6508 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±9.8%
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

The benchmark used the registered synthetic verifier_portfolio track rather than a large language-model reasoning task; verifier calls were represented by exact labels. The graph used input-channel cosine similarity within training batches rather than learned hidden-state traces, and wall-clock inference-cost savings were not measured.

How to run: python3 spectral_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

The registered verifier_portfolio benchmark was completed with identical mlp_tiny systems, paired seeds, and parity-matched learning-rate grids. The best spectral-anchor idea configuration was worse than the tuned baseline (error 0.16625 versus 0.1628125; paired delta +0.0034375; permutation p=0.6508), so it is not a demonstrated win.

Baseline: verifier_portfolio/mlp_tiny, best lr=0.003 and epochs=18; full 8-seed error mean 0.1628125, std 0.0140556
Idea: spectral anchor consensus, best lr=0.003, epochs=18, alpha=0.03; 8-seed error mean 0.16625, std 0.0249687; paired delta +0.0034375; p=0.6508

Benchmark result