△ Mechanism confirmed, baseline not beaten 2026

Distributed E-Value Prediction Sets

Implementation & benchmark of arXiv:2607.14906 — Finite-Sample Conformal Coverage Recovery via Fusion under Degraded Local Guarantees in Occupancy Map Estimation

Usefulness7/10
Difficulty5/10
Novelty5/10

Source paper: Finite-Sample Conformal Coverage Recovery via Fusion under Degraded Local Guarantees in Occupancy Map Estimation arXiv:2607.14906 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Equip each neural-network expert or robot with a locally calibrated e-value for every candidate label, then fuse neighboring e-values using uncertainty-attenuated convex weights. At inference time, retain all labels whose fused e-value does not cross the finite-sample rejection threshold, so the model abstains instead of making an unsupported point prediction. This transfers the paper's coverage-recovery mechanism to ensembles, federated models, and graph neural networks.

Formulas

$$e_i(x,y)\ge 0,\qquad \mathbb{E}[e_i(X,Y)]\le 1$$
$$w_i(x)=\frac{\exp[-\kappa u_i(x)]}{\sum_{j\in S(x)}\exp[-\kappa u_j(x)]},\qquad e_F(x,y)=\sum_{i\in S(x)}w_i(x)e_i(x,y)$$
$$C_\alpha(x)=\left\{y:e_F(x,y)<\frac{1}{\alpha}\right\},\qquad \Pr(Y\notin C_\alpha(X))\le \alpha$$
$$\Pr\left(e_F(X,Y)\ge\frac{1}{\alpha}\right)\le \alpha\,\mathbb{E}[e_F(X,Y)]\le\alpha$$

Mathematical statement

For candidate label y, expert i produces a nonnegative e-value e_i(x,y), where x is the input. Under the null hypothesis that y is the true label, local validity means E[e_i(X,Y)] <= 1. Let w_i(x) be nonnegative uncertainty-attenuation weights over the active neighborhood S(x), with sum_i w_i(x)=1. The fused e-value is e_F(x,y)=sum_i w_i(x)e_i(x,y). If the weights are fixed from calibration-time information or are otherwise valid under the null, then E[e_F(X,Y)] <= 1 without requiring independence between experts. Markov's inequality gives P(e_F(X,Y) >= 1/alpha) <= alpha, so the prediction set C_alpha(x)={y:e_F(x,y)<1/alpha} has coverage at least 1-alpha. A possible attenuation rule is w_i(x)=exp(-kappa u_i(x))/sum_j exp(-kappa u_j(x)), where u_i(x) is a calibrated uncertainty score and kappa is nonnegative.

Implementation notes

1. Integration point: use an ensemble of M neural classifiers, federated clients, or graph-neural-network nodes. Each model outputs logits, probabilities, and a scalar uncertainty u_i(x), such as entropy or calibrated energy. Reserve a calibration split and convert each model's nonconformity score into a locally valid e-value. A simple MVP is e_i(x,y)=exp(-lambda s_i(x,y))/Z_i, with the normalizer Z_i selected on calibration data so the held-out mean e-value for each label is at most one. 2. Pseudocode: for each input x, collect outputs from active neighbors; compute w_i=softmax(-kappa u_i); for every label y compute e_F(y)=sum_i w_i e_i(y); return C={y:e_F(y)<1/alpha}; abstain if C contains multiple labels or is empty. 3. Computed directly from the mechanism are nonnegative e-values, convex fusion, and the 1/alpha threshold. Estimated empirically are calibration normalizers, uncertainty scores, and graph effects. Avoid weights trained on the same target label without recalibration, because data-dependent weights can invalidate the expectation bound. 4. First cheap experiment: train five small MLPs or CNNs on CIFAR-10, calibrate each on a held-out split, and compare one model, unweighted fusion, and uncertainty-weighted fusion at alpha=0.1. Measure marginal coverage, average set size, and abstention while varying the graph from disconnected to complete. The prediction is coverage at least 0.90 up to sampling error for every graph and noise distribution, while denser graphs reduce ambiguous prediction sets. The empirical rejection frequency should track the Markov boundary e_F>=10. Falsification occurs if coverage is below 0.90 despite valid local calibration or if abstention does not generally decrease with neighborhood size.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built and ran a fixed-seed distributed e-value ensemble MVP with calibration normalizers, uncertainty-attenuated convex fusion, and prediction-set inference. Fused empirical mean e-values matched the convex weighted prediction to at most 1.1e-16 across all kappa values, coverage stayed above 0.9995 at alpha=0.1, and increasing kappa reduced average set size from 3.57 to 2.84 while increasing singleton rate from 0.052 to 0.161. The mechanism manifested, but the idea did not beat the strongest-expert baseline on set size at alpha=0.1.

Agent confidence
8/10
Baseline
Strongest expert: point accuracy 0.9659, e-value coverage 0.99994, average set size 2.92814. Unweighted fusion: coverage 0.99954, average set size 3.56554, singleton rate 0.05208.
Idea
Uncertainty-weighted fusion at kappa=2: coverage 0.99996, rejection 0.00004 versus Markov bound 0.09926, average set size 3.06882, singleton rate 0.11758, point accuracy 0.99810. Kappa 0 to 5 reduced set size from 3.56554 to 2.83658.

Limitations: This is a synthetic toy ensemble, not CIFAR-10, federated learning, or a graph neural network. Calibration validity is empirical with a 1% safety factor rather than a formal finite-sample construction; uncertainty weights are fixed externally, and learned data-dependent weights, distribution shift, repeated random seeds, runtime, communication cost, and larger neural models were not tested.

How to run: python3 distributed_evalue_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.14906)

No significant difference vs. the tuned baseline (+0.1%, p=0.7525, wins 1 / 8 paired seeds).

Benchmark
Vision (image classification)
Model
cnn_small
Paired seeds
8
Baseline mean
0.8838
Idea mean
0.8844
Effect (Δ)
0.0006 (+0.1%; negative = idea better)
Wins
1 / 8 paired seeds
p-value
0.7525 (permutation test, 20 000 shuffles)
Smallest detectable effect
±0.2%
Mechanism
Confirmed ✓
Practical effect
No effect
Baseline tuning
swept over 3 configs
Limitations:

The compact run used 400 training and 400 test CIFAR examples, 3 epochs, and two experts. Uncertainty was mean calibration entropy rather than a separately calibrated uncertainty model; formal finite-sample validity under adaptive weights and distribution shift was not tested; prediction sets were vacuous at alpha=0.1.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark latest Failed ✗

Agent confidence: 9/10

Built and evaluated distributed e-value prediction sets on the registered vision/cnn_small benchmark with held-out local calibration, entropy-attenuated convex fusion, an equal-budget baseline sweep, and 8 paired seeds. The mechanism signature was quantitatively confirmed, but the idea had slightly higher classification error than the tuned baseline and no significant improvement; its alpha=0.1 prediction sets were vacuous.

Baseline
Test error mean 0.88375 across 8 seeds; sweep means lr=0.001: 0.88000, lr=0.003: 0.88625, lr=0.01: 0.89500.
Idea
Test error mean 0.884375 across 8 seeds at lr=0.001; paired delta +0.000625, permutation p-value 0.7525, 1/8 paired wins; coverage 1.0 and average set size 10.0.

Benchmark result

No significant difference vs. the tuned baseline (+0.1%, p=0.7525, wins 1 / 8 paired seeds).

Benchmark
Vision (image classification)
Model
cnn_small
Paired seeds
8
Baseline mean
0.8838
Idea mean
0.8844
Effect (Δ)
0.0006 (+0.1%; negative = idea better)
Wins
1 / 8 paired seeds
p-value
0.7525 (permutation test, 20 000 shuffles)
Smallest detectable effect
±0.2%
Mechanism
Confirmed ✓
Practical effect
No effect
Baseline tuning
swept over 3 configs
Limitations:

The compact run used 400 training and 400 test CIFAR examples, 3 epochs, and two experts. Uncertainty was mean calibration entropy rather than a separately calibrated uncertainty model; formal finite-sample validity under adaptive weights and distribution shift was not tested; prediction sets were vacuous at alpha=0.1.

How to run: python3 bench_experiment.py

Artifacts

Implementation overview ⬇ Download all as ZIP 6 files · code, reports and structured results