Spatial-Quantile Conformal Bands for Neural Operators
Source paper: Conformal Uncertainty Quantification Guarantees for Neural Operators arXiv:2608.28515 ⓘ · analyzed Sep 2, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace a worst-case spatial residual score with the (1-gamma)-quantile of the normalized residual field, then calibrate this scalar score on held-out operator examples. At test time, inflate the predicted uncertainty field by the conformal order statistic; the guarantee targets the fraction of spatial domain covered, producing tighter bands than max-error or Bonferroni corrections.
Formulas
Mathematical statement
For an input function a in A, let u^dagger(a)(x)=G^dagger(a)(x) be the true field, hat u(a)(x)=hat G(a)(x) the neural-operator prediction, and hat e(a)(x)=hat E(a)(x)>0 a scale field. With domain probability measure nu satisfying nu(D)=1, define the normalized residual r_a(x)=||u^dagger(a)(x)-hat u(a)(x)||_2/hat e(a)(x). The spatial score is S(a)=inf{s>=0: nu({x in D:r_a(x)<=s})>=1-gamma}. Given n exchangeable calibration pairs, sort S_i=S(a_i) as S_(1)<=...<=S_(n), and choose q=S_(k), where k=min{n,ceil((n+1)(1-alpha))}. The band is C_q(a)(x)={v in R^{d_u}:||v-hat u(a)(x)||_2<=q hat e(a)(x)}. The paper's measurable split-conformal result gives P[nu({x:G^dagger(a)(x) in C_q(a)(x)})>=1-gamma] >= 1-alpha, with probability over a and calibration data, under exchangeability and measurable field representatives. On a grid, nu is replaced by normalized cell or quadrature weights.
Implementation notes
1. Integrate the method after a trained neural operator, such as an FNO or DeepONet, at inference time. The model must output a field hat u on the target grid. Add a positive uncertainty head hat e=softplus(h)+epsilon, or use a validation-derived residual scale map; conformal calibration itself is not backpropagated. 2. Pseudocode: for every held-out calibration pair (a_i,u_i), compute predictions hat u_i and scales hat e_i; calculate r_ij=||u_ij-hat u_ij||_2/(hat e_ij+epsilon); sort r_ij over spatial cells and find the weighted empirical quantile containing mass 1-gamma, producing S_i. Sort the S_i values, set k=min(n,ceil((n+1)(1-alpha))), and set q=S_(k). For a test input, return center hat u_j and radius q*hat e_j at each cell. 3. Compute spatial weights from uniform cells or quadrature weights normalized to sum one. Estimate the scale field from the uncertainty head or training-only residual statistics, and apply identical positivity, clipping, and interpolation rules during calibration and testing. 4. First experiment: train a small FNO on 2-D Darcy flow with disjoint training, calibration, and test sets. Compare against max-residual conformal calibration, pointwise split conformal, and Bonferroni intervals using identical model weights, calibration size, grid, and inference cost; evaluate alpha=0.1 and gamma in {0.01,0.05,0.10}. 5. Pre-register tests: repeated test batches must achieve domain coverage at least 1-gamma in at least 1-alpha of batches, allowing tolerance 0.02; at matched coverage, average band width should be at least 20% smaller than max-residual calibration; increasing gamma must monotonically decrease mean radius; and increasing calibration size must reduce coverage shortfall below 0.02. Measure weighted spatial hit fractions, not pointwise hit rates. The decisive ablation replaces the spatial-quantile score with max_j r_ij. Falsify the transfer if coverage is systematically below target, spatial quantiles do not tighten bands at matched coverage, or nonuniform quadrature weights break calibration.
Verification
Stage 1 · Toy mechanism gate: Passed ✓
Stage 2 · Mechanism transferred to benchmark: Confirmed ✓
Stage 2 · Practical benchmark result: No measurable effect
Stage 1 — Mechanism check agent confidence 8/10
Built spatial-quantile conformal calibration with weighted spatial order statistics, finite-sample calibration indexing, positive frozen scale fields, and a reproducible synthetic operator experiment. The math sanity checks passed, including weighted quantile behavior and monotonic tightening as gamma increases. On the mini-experiment, spatial-quantile bands were 5.3% narrower at gamma=0.05 and 11.8% narrower at gamma=0.10 than max-residual bands, while the >=(1-alpha) domain-coverage event rates were 0.903 and 0.906 respectively; the effect disappeared at gamma=0.01 because the 48-cell spatial quantile equals the maximum at that resolution.
- Agent confidence
- 8/10
- Baseline
- Max-residual conformal: q=0.0041301 for all gamma; mean width=0.0002891; event rates gamma=.01/.05/.10 = 0.909/0.917/0.930; mean spatial hit fractions = 0.977/0.977/0.977
- Idea
- Spatial-quantile conformal: q=.0041301/.0039117/.0036424 for gamma=.01/.05/.10; mean widths=.0002891/.0002738/.0002550; event rates=.909/.903/.906; mean spatial hit fractions=.977/.973/.967
Limitations: This is a small synthetic smooth-operator surrogate rather than a trained FNO on 2-D Darcy flow. Only one fixed train/calibration/test split was run; repeated calibration-size studies, Bonferroni and pointwise baselines, and a nonuniform-weight end-to-end coverage experiment were not performed.
How to run: python3 spatial_quantile_conformal.py
Stage 2 — Benchmark vs. tuned baseline
No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 8 paired seeds).
- Benchmark
- Poisson_field_operator
- Model
- mlp_tiny
- Paired seeds
- 8
- Baseline mean
- 0.0014
- Idea mean
- 0.0014
- Effect (Δ)
- 0 (−-0.0%; negative = idea better)
- Wins
- 0 / 8 paired seeds
- p-value
- 1 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- n/a
- Mechanism
- Confirmed ✓
- Practical effect
- No effect
- Baseline tuning
- swept over 3 configs
The custom task is a small manufactured 2-D Poisson/BVP surrogate rather than a full Darcy/FNO benchmark. Only gamma=0.10 was used in the final 8-seed bench; no calibration-size, nonuniform-weight, Bonferroni, or pointwise conformal ablations were run.
How to run: python3 math_check.py && /home/maxwelhelp/main/bin/python3 run_bench.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Stage 2 — Benchmark latest Failed ✗
Agent confidence: 9/10
Built a local manufactured Poisson/BVP operator track with independently trained matched MLP systems, max-residual conformal baseline, and spatial-quantile conformal bands. The numerical check passed; trained-model bands were 37.3% narrower with approximately 0.898 domain-event rate versus 0.996 for the baseline. Primary test MSE was exactly unchanged across all paired seeds, with delta_mean=0 and permutation p=1.0, so there was no learning win.
- Baseline
- MSE mean 0.00144131, std 0.00008870, best lr 0.006; mean band width 0.249315; domain-event rate 0.99625.
- Idea
- MSE mean 0.00144131, std 0.00008870, best lr 0.006, gamma 0.10; mean band width 0.156211; domain-event rate 0.898125; paired delta 0, p=1.0.
Benchmark result
No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 8 paired seeds).
- Benchmark
- Poisson_field_operator
- Model
- mlp_tiny
- Paired seeds
- 8
- Baseline mean
- 0.0014
- Idea mean
- 0.0014
- Effect (Δ)
- 0 (−-0.0%; negative = idea better)
- Wins
- 0 / 8 paired seeds
- p-value
- 1 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- n/a
- Mechanism
- Confirmed ✓
- Practical effect
- No effect
- Baseline tuning
- swept over 3 configs
The custom task is a small manufactured 2-D Poisson/BVP surrogate rather than a full Darcy/FNO benchmark. Only gamma=0.10 was used in the final 8-seed bench; no calibration-size, nonuniform-weight, Bonferroni, or pointwise conformal ablations were run.
How to run: python3 math_check.py && /home/maxwelhelp/main/bin/python3 run_bench.py
Artifacts
- bench_report.json 5.4 KB View Raw JSON
- math_check.py 1.0 KB View Raw JSON
- poisson_field_track.py 1.4 KB View Raw JSON
- report.md 1.6 KB View
- report_bench_2026-09-02T121801.md 3.4 KB View
- results.json 1.5 KB View Raw JSON
- run_bench.py 4.2 KB View Raw JSON
- spatial_quantile_conformal.py 5.1 KB View Raw JSON