# Pick-to-Learn Scenario Compression for Safe NN Calibration

- ID: 999
- Canonical URL: https://synthcore.org/idea/999/pick-to-learn-scenario-compression-for-safe-nn-calibration
- API JSON: https://synthcore.org/api/idea/999.json
- API Markdown: https://synthcore.org/api/idea/999.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.16084](https://arxiv.org/abs/2607.16084)
- Category: training
- Solves: generalization, stability, sample-efficiency
- ML areas: optimizer, training-dynamics, regularization, rl
- Math tags: probability, statistics, optimization, control-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace uniform tuning of neural-network hyperparameters with a Pick-to-Learn-style compression procedure that selects the few scenarios most informative for constraint satisfaction. A scenario can be a domain-randomization seed, adversarial perturbation, task instance, or rollout. Tune the network or optimizer on the selected compression set, then evaluate fresh scenarios using a finite-sample certificate for the probability of violating a prescribed robustness, safety, or stability constraint.

## Mathematical statement

Let xi denote an iid uncertainty realization, such as an environment, perturbation, or task instance. Let D_N = {xi_1, ..., xi_N} be N observed scenarios, theta be neural-network and training-policy parameters, and g(theta, xi) <= 0 be the required scenario constraint. Pick-to-Learn constructs a data-dependent compression set C(D_N) contained in D_N with size k much smaller than N, and calibrates theta using the selected informative scenarios. The fresh-scenario violation probability is V(theta) = P_{xi approximately distributed as P}[g(theta, xi) > 0]. The scenario-compression guarantee has the form P_{D_N}{V(theta(D_N)) <= epsilon(N,k,beta)} >= 1 - beta. In the paper's reported case, N = 400, k = 2, beta = 10^{-5}, and epsilon = 0.048, so the certified probability of violating the constraint on a fresh scenario is at most 4.8 percent with confidence 1 - 10^{-5}. The useful mechanism is that a small set of active or informative scenarios can preserve an out-of-sample constraint guarantee, provided the scenarios are iid and the calibration procedure satisfies the assumptions of the theorem.

## Key formulas

- $$C(D_N)\subseteq D_N,\qquad |D_N|=N=400,\qquad |C(D_N)|=k=2.$$
- $$V(\theta)=\Pr_{\xi\sim P}\left[g(\theta,\xi)>0\right].$$
- $$\Pr_{D_N}\left\{V\left(\theta(D_N)\right)\leq \varepsilon(N,k,\beta)\right\}\geq 1-\beta.$$
- $$\varepsilon(400,2,10^{-5})=0.048,\qquad \beta=10^{-5}.$$

## Implementation notes

First, place this method around an existing neural-network training and hyperparameter-calibration loop. Let theta include learning rate, weight decay, dropout, spectral-normalization strength, controller gains, or a small set of trainable policy parameters. Define g(theta, xi) as a measurable constraint such as validation loss minus a threshold, maximum rollout deviation, gradient-norm explosion, safety violation, or failure of a certified Lipschitz bound. Generate N iid scenarios and initialize an empty compression set. A practical greedy algorithm is: train or calibrate theta on the current set; score every unselected scenario by g(theta, xi); select the scenario with the largest positive violation or largest change in the active constraint set; add it to the compression set; recalibrate theta; repeat until k scenarios are selected or no scenario changes the active set. Finally retrain on the selected set and evaluate all N scenarios for diagnostics. The compression set and training constraints are computed from sampled data, while the fresh violation probability is estimated only with an independent test set. The exact epsilon(N,k,beta) should be taken from the full Pick-to-Learn theorem rather than replaced by a heuristic confidence interval. The first cheap experiment should use an MLP on MNIST-C with 400 corruption scenarios and a fixed corruption-loss threshold. Compare greedy selection of k = 2, 4, and 8 scenarios against random selection with the same retraining budget, then evaluate on 10,000 fresh corruptions. The predicted signature is that greedy compression reaches near-complete training-scenario coverage with far fewer scenarios, and for N = 400, k = 2, beta = 10^{-5}, the fresh violation rate should be near or below 4.8 percent when the theorem assumptions hold. Non-iid scenarios or uncontrolled adaptive tuning should invalidate or weaken this boundary.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built a readable Pick-to-Learn MVP with exact theorem-bound evaluation, greedy largest-violation compression, constrained calibration, and iid fresh-scenario diagnostics. The mechanism manifested: epsilon decreased roughly by half when N doubled, increased with weaker beta and larger k, and greedy k=2 achieved 0.774% fresh violation versus 54.162% for random k=2 in the toy problem. The paper’s displayed equation gives 3.53% for N=400,k=2,beta=1e-5, differing from its quoted 4.8%; this implementation reports the literal equation result rather than silently using 4.8%.

### Mechanism check

- Verdict: Built a readable Pick-to-Learn MVP with exact theorem-bound evaluation, greedy largest-violation compression, constrained calibration, and iid fresh-scenario diagnostics. The mechanism manifested: epsilon decreased roughly by half when N doubled, increased with weaker beta and larger k, and greedy k=2 achieved 0.774% fresh violation versus 54.162% for random k=2 in the toy problem. The paper’s displayed equation gives 3.53% for N=400,k=2,beta=1e-5, differing from its quoted 4.8%; this implementation reports the literal equation result rather than silently using 4.8%.
- Confidence: 8/10
- Limitations: The benchmark is a two-parameter affine convex surrogate, not an MLP on MNIST-C, MPC, or rollout calibration. Only one seed and one random-compression draw were used; no formal theorem claim is made for adaptive neural-network training, and the shifted-distribution diagnostic is not a certificate test.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_p2l.py](https://synthcore.org/code/412/bench_p2l.py)
- [bench_report.json](https://synthcore.org/code/412/bench_report.json)
- [p2l_experiment.py](https://synthcore.org/code/412/p2l_experiment.py)
- [report.md](https://synthcore.org/code/412/report.md)
- [report_bench_2026-08-31T114019.md](https://synthcore.org/code/412/report_bench_2026-08-31T114019.md)
- [results.json](https://synthcore.org/code/412/results.json)
- [Download all files as ZIP](https://synthcore.org/download/412)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.