Mechanism confirmed, baseline not beaten 2026

Spatial-Quantile Conformal Bands for Neural Operators

Usefulness7/10
Difficulty3/10
Novelty5/10

Source paper: Conformal Uncertainty Quantification Guarantees for Neural Operators arXiv:2608.28515 · analyzed Sep 2, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a worst-case spatial residual score with the (1-gamma)-quantile of the normalized residual field, then calibrate this scalar score on held-out operator examples. At test time, inflate the predicted uncertainty field by the conformal order statistic; the guarantee targets the fraction of spatial domain covered, producing tighter bands than max-error or Bonferroni corrections.

Formulas

$$r_a(x)=\frac{\|\mathcal G^\dagger(a)(x)-\hat{\mathcal G}(a)(x)\|_2}{\hat{\mathcal E}(a)(x)},\qquad S(a)=\inf\left\{s\geq0:\nu\left(\left\{x\in D:r_a(x)\leq s\right\}\right)\geq1-\gamma\right\}.$$
$$k=\min\left\{n,\left\lceil(n+1)(1-\alpha)\right\rceil\right\},\qquad q=S_{(k)},\quad S_{(1)}\leq\cdots\leq S_{(n)}.$$
$$C_q(a)(x)=\left\{v\in\mathbb R^{d_u}:\|v-\hat{\mathcal G}(a)(x)\|_2\leq q\,\hat{\mathcal E}(a)(x)\right\}.$$
$$\Pr\left[\nu\left(\left\{x\in D:\mathcal G^\dagger(a)(x)\in C_q(a)(x)\right\}\right)\geq1-\gamma\right]\geq1-\alpha.$$

Mathematical statement

For an input function a in A, let u^dagger(a)(x)=G^dagger(a)(x) be the true field, hat u(a)(x)=hat G(a)(x) the neural-operator prediction, and hat e(a)(x)=hat E(a)(x)>0 a scale field. With domain probability measure nu satisfying nu(D)=1, define the normalized residual r_a(x)=||u^dagger(a)(x)-hat u(a)(x)||_2/hat e(a)(x). The spatial score is S(a)=inf{s>=0: nu({x in D:r_a(x)<=s})>=1-gamma}. Given n exchangeable calibration pairs, sort S_i=S(a_i) as S_(1)<=...<=S_(n), and choose q=S_(k), where k=min{n,ceil((n+1)(1-alpha))}. The band is C_q(a)(x)={v in R^{d_u}:||v-hat u(a)(x)||_2<=q hat e(a)(x)}. The paper's measurable split-conformal result gives P[nu({x:G^dagger(a)(x) in C_q(a)(x)})>=1-gamma] >= 1-alpha, with probability over a and calibration data, under exchangeability and measurable field representatives. On a grid, nu is replaced by normalized cell or quadrature weights.

Implementation notes

1. Integrate the method after a trained neural operator, such as an FNO or DeepONet, at inference time. The model must output a field hat u on the target grid. Add a positive uncertainty head hat e=softplus(h)+epsilon, or use a validation-derived residual scale map; conformal calibration itself is not backpropagated. 2. Pseudocode: for every held-out calibration pair (a_i,u_i), compute predictions hat u_i and scales hat e_i; calculate r_ij=||u_ij-hat u_ij||_2/(hat e_ij+epsilon); sort r_ij over spatial cells and find the weighted empirical quantile containing mass 1-gamma, producing S_i. Sort the S_i values, set k=min(n,ceil((n+1)(1-alpha))), and set q=S_(k). For a test input, return center hat u_j and radius q*hat e_j at each cell. 3. Compute spatial weights from uniform cells or quadrature weights normalized to sum one. Estimate the scale field from the uncertainty head or training-only residual statistics, and apply identical positivity, clipping, and interpolation rules during calibration and testing. 4. First experiment: train a small FNO on 2-D Darcy flow with disjoint training, calibration, and test sets. Compare against max-residual conformal calibration, pointwise split conformal, and Bonferroni intervals using identical model weights, calibration size, grid, and inference cost; evaluate alpha=0.1 and gamma in {0.01,0.05,0.10}. 5. Pre-register tests: repeated test batches must achieve domain coverage at least 1-gamma in at least 1-alpha of batches, allowing tolerance 0.02; at matched coverage, average band width should be at least 20% smaller than max-residual calibration; increasing gamma must monotonically decrease mean radius; and increasing calibration size must reduce coverage shortfall below 0.02. Measure weighted spatial hit fractions, not pointwise hit rates. The decisive ablation replaces the spatial-quantile score with max_j r_ij. Falsify the transfer if coverage is systematically below target, spatial quantiles do not tighten bands at matched coverage, or nonuniform quadrature weights break calibration.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Stage 1 — Mechanism check agent confidence 8/10

Built spatial-quantile conformal calibration with weighted spatial order statistics, finite-sample calibration indexing, positive frozen scale fields, and a reproducible synthetic operator experiment. The math sanity checks passed, including weighted quantile behavior and monotonic tightening as gamma increases. On the mini-experiment, spatial-quantile bands were 5.3% narrower at gamma=0.05 and 11.8% narrower at gamma=0.10 than max-residual bands, while the >=(1-alpha) domain-coverage event rates were 0.903 and 0.906 respectively; the effect disappeared at gamma=0.01 because the 48-cell spatial quantile equals the maximum at that resolution.

Agent confidence
8/10
Baseline
Max-residual conformal: q=0.0041301 for all gamma; mean width=0.0002891; event rates gamma=.01/.05/.10 = 0.909/0.917/0.930; mean spatial hit fractions = 0.977/0.977/0.977
Idea
Spatial-quantile conformal: q=.0041301/.0039117/.0036424 for gamma=.01/.05/.10; mean widths=.0002891/.0002738/.0002550; event rates=.909/.903/.906; mean spatial hit fractions=.977/.973/.967

Limitations: This is a small synthetic smooth-operator surrogate rather than a trained FNO on 2-D Darcy flow. Only one fixed train/calibration/test split was run; repeated calibration-size studies, Bonferroni and pointwise baselines, and a nonuniform-weight end-to-end coverage experiment were not performed.

How to run: python3 spatial_quantile_conformal.py

Stage 2 — Benchmark vs. tuned baseline

No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 8 paired seeds).

Benchmark
Poisson_field_operator
Model
mlp_tiny
Paired seeds
8
Baseline mean
0.0014
Idea mean
0.0014
Effect (Δ)
0 (−-0.0%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
1 (permutation test, 20 000 shuffles)
Smallest detectable effect
n/a
Mechanism
Confirmed ✓
Practical effect
No effect
Baseline tuning
swept over 3 configs
Limitations:

The custom task is a small manufactured 2-D Poisson/BVP surrogate rather than a full Darcy/FNO benchmark. Only gamma=0.10 was used in the final 8-seed bench; no calibration-size, nonuniform-weight, Bonferroni, or pointwise conformal ablations were run.

How to run: python3 math_check.py && /home/maxwelhelp/main/bin/python3 run_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark latest Failed ✗

Agent confidence: 9/10

Built a local manufactured Poisson/BVP operator track with independently trained matched MLP systems, max-residual conformal baseline, and spatial-quantile conformal bands. The numerical check passed; trained-model bands were 37.3% narrower with approximately 0.898 domain-event rate versus 0.996 for the baseline. Primary test MSE was exactly unchanged across all paired seeds, with delta_mean=0 and permutation p=1.0, so there was no learning win.

Baseline
MSE mean 0.00144131, std 0.00008870, best lr 0.006; mean band width 0.249315; domain-event rate 0.99625.
Idea
MSE mean 0.00144131, std 0.00008870, best lr 0.006, gamma 0.10; mean band width 0.156211; domain-event rate 0.898125; paired delta 0, p=1.0.

Benchmark result

No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 8 paired seeds).

Benchmark
Poisson_field_operator
Model
mlp_tiny
Paired seeds
8
Baseline mean
0.0014
Idea mean
0.0014
Effect (Δ)
0 (−-0.0%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
1 (permutation test, 20 000 shuffles)
Smallest detectable effect
n/a
Mechanism
Confirmed ✓
Practical effect
No effect
Baseline tuning
swept over 3 configs
Limitations:

The custom task is a small manufactured 2-D Poisson/BVP surrogate rather than a full Darcy/FNO benchmark. Only gamma=0.10 was used in the final 8-seed bench; no calibration-size, nonuniform-weight, Bonferroni, or pointwise conformal ablations were run.

How to run: python3 math_check.py && /home/maxwelhelp/main/bin/python3 run_bench.py

Artifacts

⬇ Download all as ZIP 8 files · code, reports and structured results