Uncertainty-guided family sampling
Implementation & benchmark of arXiv:2607.15060 — Learning the Fermion sign structure in path-integral Monte Carlo
Source paper: Learning the Fermion sign structure in path-integral Monte Carlo arXiv:2607.15060 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Use the family predictor not only as a post-processing estimator but also as a feedback controller for data collection. Reweight Monte Carlo proposals or minibatch selection toward under-sampled families whose signed contribution and predictive uncertainty are large, rather than spending samples on already well-known positive families.
Formulas
Mathematical statement
For family c, let a_c=s_cP_cE_c be its signed contribution, u_c^2 the estimated predictive variance of that contribution, and n_c its current sample count. The allocation mechanism increases sampling of families with large uncertainty-weighted contribution using q_c proportional to (|a_c|+kappa)u_c/sqrt(n_c), with kappa>0 preventing zero-contribution families from being ignored. Under approximately independent family-conditioned noise, the numerator variance is Var(Nhat) approximately sum_c u_c^2/n_c for N=sum_c a_c. An exploration mixture q'_c=(1-epsilon)q_c+epsilon/|C| prevents mode collapse.
Implementation notes
1. Integration point: attach the controller to an active-learning loop, replay buffer, or Monte Carlo proposal kernel. Maintain one reservoir per family and periodically update sampling priorities from the current neural predictor. For a neural network, dropout or an ensemble of K heads supplies u_c^2; for a Monte Carlo chain, use the score to bias proposals while retaining importance weights so the target distribution remains unchanged. 2. Pseudocode: initialize uniform exploration; collect samples and fit the family-residual model; for each family compute the ensemble mean of s_cP_cE_c and its variance u_c^2; form q_c^mix using the displayed score; draw the next batch or choose proposals from this distribution; apply inverse-proposal weights when estimating training moments; refit and repeat. 3. Computed from the paper’s mechanism: family-wise active importance sampling directed toward under-sampled permutation regions. Estimated empirically: P_c, E_c, u_c, effective sample size, and acceptance rates. 4. First cheap experiment: create a synthetic signed-family regression problem with 32 families, exponentially small P_c for rare families, and known family energies. Compare uniform sampling, magnitude-only prioritization, and uncertainty-weighted sampling at equal budget. The prediction is that active sampling reduces numerator variance according to sum_c u_c^2/n_c and reaches a fixed energy error with at least a twofold lower budget. Sweep epsilon; performance should deteriorate as epsilon approaches zero if a family is initially mispredicted, providing a direct mode-collapse test.
Verification
Stage 1 · Toy mechanism gate: Passed ✓
Stage 2 · Mechanism transferred to benchmark: Confirmed ✓
Stage 2 · Practical benchmark result: No measurable effect
Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works
Stage 1 — Mechanism check agent confidence 8/10
Built and numerically tested an uncertainty-guided family allocation controller with exploration mixing. The variance identity matched Monte Carlo within 2.3%, allocation proportional to u reduced predicted variance by a stable 1.46x across budgets, and exploration prevented severe mode collapse when one high-variance family was deliberately mispredicted. The controller achieved predicted/empirical variance 0.899/0.893 versus 1.258/1.291 for uniform and 0.913/0.917 for magnitude-only sampling, though its gain over magnitude-only was modest.
- Agent confidence
- 8/10
- Baseline
- Uniform: predicted variance 1.2576, empirical variance 1.2911; magnitude-only: predicted 0.9129, empirical 0.9170.
- Idea
- Uncertainty-guided controller: predicted variance 0.8986, empirical variance 0.8928; proportional-allocation sweep gave 1.4628x uniform variance reduction at budget 4096. Under hidden high-uncertainty family, epsilon 0.0/0.01/0.05/0.2 yielded family counts 3/5/10/33 and true predicted variances 8.97/5.68/3.22/1.51.
Limitations: This is a synthetic independent-Gaussian family model with known uncertainties, not a neural predictor or real permutation-family Monte Carlo system. It does not test learned uncertainty calibration, importance-weight correction, correlated family noise, estimator bias, wall-clock efficiency, or a full training/active-learning loop.
How to run: python3 experiment.py
Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.15060)
No significant difference vs. the tuned baseline (+1.4%, p=0.6194, wins 4 / 8 paired seeds).
- Benchmark
- Tabular regression (Friedman)
- Model
- mlp_tiny
- Paired seeds
- 8
- Baseline mean
- 9.4814
- Idea mean
- 9.6101
- Effect (Δ)
- 0.1286 (+1.4%; negative = idea better)
- Wins
- 4 / 8 paired seeds
- p-value
- 0.6194 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- n/a
- Mechanism
- Confirmed ✓
- Practical effect
- No effect
- Baseline tuning
- swept over 0 configs
Only tabular was tested. Families were fixed quantile bins of the first feature; uncertainty was estimated from within-family residual spread rather than ensembles or dropout; the run used 400 training examples, 200 test examples, 12 epochs, and batch size 64. Larger datasets, other tracks, wall-clock efficiency, correlated family noise, and importance-weight variance were not tested.
How to run: python3 stage2_family_bench.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Stage 2 — Benchmark latest Failed ✗
Agent confidence: 9/10
The uncertainty-guided family sampler was implemented and tested on the structurally matched tabular Friedman regression track with identical mlp_tiny systems and equal training budgets. Its trained-model allocation-variance prediction was confirmed, but test MSE was slightly worse than the tuned uniform baseline: 9.6101 versus 9.4814, paired delta +0.1286, permutation p=0.61945. Therefore the idea does not give a significant benchmark win.
- Baseline
- lr=0.006; test MSE mean 9.4814206362, std 0.9072998646.
- Idea
- lr=0.006, epsilon=0.3; test MSE mean 9.6100564003, std 1.0356528913; 4/8 paired wins.
- Mechanism signature
- Predicted allocation variance 0.0982159125, observed proxy 0.0913693272, relative error 6.97%, confirmed=true.
Benchmark result
No significant difference vs. the tuned baseline (+1.4%, p=0.6194, wins 4 / 8 paired seeds).
- Benchmark
- Tabular regression (Friedman)
- Model
- mlp_tiny
- Paired seeds
- 8
- Baseline mean
- 9.4814
- Idea mean
- 9.6101
- Effect (Δ)
- 0.1286 (+1.4%; negative = idea better)
- Wins
- 4 / 8 paired seeds
- p-value
- 0.6194 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- n/a
- Mechanism
- Confirmed ✓
- Practical effect
- No effect
- Baseline tuning
- swept over 0 configs
Only tabular was tested. Families were fixed quantile bins of the first feature; uncertainty was estimated from within-family residual spread rather than ensembles or dropout; the run used 400 training examples, 200 test examples, 12 epochs, and batch size 64. Larger datasets, other tracks, wall-clock efficiency, correlated family noise, and importance-weight variance were not tested.
How to run: python3 stage2_family_bench.py
Artifacts
- bench_report.json 4.2 KB View Raw
- experiment.py 5.2 KB View Raw
- report.md 1.6 KB View
- report_bench_2026-08-31T111901.md 3.0 KB View
- results.json 3.1 KB View Raw
- stage2_family_bench.py 6.6 KB View Raw