# Uncertainty-guided family sampling

- ID: 947
- Canonical URL: https://synthcore.org/idea/947/uncertainty-guided-family-sampling
- API JSON: https://synthcore.org/api/idea/947.json
- API Markdown: https://synthcore.org/api/idea/947.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.15060](https://arxiv.org/abs/2607.15060)
- Category: training
- Solves: sample-efficiency, accuracy, stability
- ML areas: optimizer, training-dynamics, data-augmentation, rl
- Math tags: probability, statistics, optimization, stochastic-processes
- Ratings: usefulness 7/10; difficulty 6/10; novelty 5/10

## Idea description

Use the family predictor not only as a post-processing estimator but also as a feedback controller for data collection. Reweight Monte Carlo proposals or minibatch selection toward under-sampled families whose signed contribution and predictive uncertainty are large, rather than spending samples on already well-known positive families.

## Mathematical statement

For family c, let a_c=s_cP_cE_c be its signed contribution, u_c^2 the estimated predictive variance of that contribution, and n_c its current sample count. The allocation mechanism increases sampling of families with large uncertainty-weighted contribution using q_c proportional to (|a_c|+kappa)u_c/sqrt(n_c), with kappa>0 preventing zero-contribution families from being ignored. Under approximately independent family-conditioned noise, the numerator variance is Var(Nhat) approximately sum_c u_c^2/n_c for N=sum_c a_c. An exploration mixture q'_c=(1-epsilon)q_c+epsilon/|C| prevents mode collapse.

## Key formulas

- $$N=\sum_c s_cP_cE_c,\qquad D=\sum_c s_cP_c,\qquad \widehat E_F=N/D.$$
- $$\operatorname{Var}(\widehat N)\approx\sum_c\frac{u_c^2}{n_c},\qquad q_c\propto \frac{(|s_cP_cE_c|+\kappa)u_c}{\sqrt{n_c}}.$$
- $$q_c^{\mathrm{mix}}=(1-\epsilon)q_c+\epsilon\,|\mathcal C|^{-1},\qquad \epsilon>0.$$
- $$\text{under fixed total sample count }n,\qquad n_c\propto u_c\text{ minimizes }\sum_c u_c^2/n_c\text{ in the independent-noise approximation}.$$

## Implementation notes

1. Integration point: attach the controller to an active-learning loop, replay buffer, or Monte Carlo proposal kernel. Maintain one reservoir per family and periodically update sampling priorities from the current neural predictor. For a neural network, dropout or an ensemble of K heads supplies u_c^2; for a Monte Carlo chain, use the score to bias proposals while retaining importance weights so the target distribution remains unchanged. 2. Pseudocode: initialize uniform exploration; collect samples and fit the family-residual model; for each family compute the ensemble mean of s_cP_cE_c and its variance u_c^2; form q_c^mix using the displayed score; draw the next batch or choose proposals from this distribution; apply inverse-proposal weights when estimating training moments; refit and repeat. 3. Computed from the paper’s mechanism: family-wise active importance sampling directed toward under-sampled permutation regions. Estimated empirically: P_c, E_c, u_c, effective sample size, and acceptance rates. 4. First cheap experiment: create a synthetic signed-family regression problem with 32 families, exponentially small P_c for rare families, and known family energies. Compare uniform sampling, magnitude-only prioritization, and uncertainty-weighted sampling at equal budget. The prediction is that active sampling reduces numerator variance according to sum_c u_c^2/n_c and reaches a fixed energy error with at least a twofold lower budget. Sweep epsilon; performance should deteriorate as epsilon approaches zero if a family is initially mispredicted, providing a direct mode-collapse test.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built and numerically tested an uncertainty-guided family allocation controller with exploration mixing. The variance identity matched Monte Carlo within 2.3%, allocation proportional to u reduced predicted variance by a stable 1.46x across budgets, and exploration prevented severe mode collapse when one high-variance family was deliberately mispredicted. The controller achieved predicted/empirical variance 0.899/0.893 versus 1.258/1.291 for uniform and 0.913/0.917 for magnitude-only sampling, though its gain over magnitude-only was modest.

### Mechanism check

- Verdict: Built and numerically tested an uncertainty-guided family allocation controller with exploration mixing. The variance identity matched Monte Carlo within 2.3%, allocation proportional to u reduced predicted variance by a stable 1.46x across budgets, and exploration prevented severe mode collapse when one high-variance family was deliberately mispredicted. The controller achieved predicted/empirical variance 0.899/0.893 versus 1.258/1.291 for uniform and 0.913/0.917 for magnitude-only sampling, though its gain over magnitude-only was modest.
- Confidence: 8/10
- Limitations: This is a synthetic independent-Gaussian family model with known uncertainties, not a neural predictor or real permutation-family Monte Carlo system. It does not test learned uncertainty calibration, importance-weight correction, correlated family noise, estimator bias, wall-clock efficiency, or a full training/active-learning loop.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.61945
- Paired wins: 4/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/400/bench_report.json)
- [experiment.py](https://synthcore.org/code/400/experiment.py)
- [report.md](https://synthcore.org/code/400/report.md)
- [report_bench_2026-08-31T111901.md](https://synthcore.org/code/400/report_bench_2026-08-31T111901.md)
- [results.json](https://synthcore.org/code/400/results.json)
- [stage2_family_bench.py](https://synthcore.org/code/400/stage2_family_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/400)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.