# Pick-to-Learn Safety Fine-Tuning

- ID: 1217
- Canonical URL: https://synthcore.org/idea/1217/pick-to-learn-safety-fine-tuning
- API JSON: https://synthcore.org/api/idea/1217.json
- API Markdown: https://synthcore.org/api/idea/1217.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.21086](https://arxiv.org/abs/2607.21086)
- Category: regularization
- Solves: stability, generalization, sample-efficiency
- ML areas: rl, world-model, training-dynamics, regularization
- Math tags: probability, statistics, control-theory, stochastic-processes, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Train a neural policy against a simulator using an adaptive constraint set formed from the worst violations, rather than uniformly averaging all rollouts. At each round, identify the trajectory with the largest normalized safety violation, add its state-time features and violation margin to a surrogate barrier or penalty model, and fine-tune the policy until the surrogate constraints are satisfied. This should reduce the gap between nominal validation risk and rare-event failure risk while concentrating simulator calls on informative trajectories.

## Mathematical statement

A rollout samples initial state $x_0\sim\mu_0$, fixed uncertain parameter $\lambda\sim\mu_\Lambda$, and Brownian noise $w_t$, then evolves according to $dx_t=f(x_t,u_t,t;\lambda)dt+Gdw_t$, where $x_t$ is the state, $u_t=\pi_\theta(x_t,t)$ is the neural policy, $f$ is the drift, $G$ is the diffusion matrix, and $\theta$ are policy parameters. Let $g_r(\tau)\le 0$ be the $r$-th safety constraint evaluated on trajectory $\tau$, and define normalized violation $v(\tau)=\max_r[g_r(\tau)/s_r]_+$ with scale $s_r>0$ and $[z]_+=\max(z,0)$. Pick-to-Learn adds the maximizer $\tau^*=\arg\max_{\tau\in\mathcal{D}_m}v(\tau)$ from the current simulator batch to the surrogate constraint set $\mathcal{C}_{m+1}=\mathcal{C}_m\cup\{(\tau^*,v(\tau^*))\}$. A conservative empirical certificate uses $\widehat p_v=N_v/N$ and an upper confidence correction $\epsilon(N,\delta,\mathcal{K})$, giving $p_v\le\widehat p_v+\epsilon$ with confidence at least $1-\delta$; $N_v$ is the number of violating test rollouts, $N$ is the test-set size, and $\mathcal{K}$ is the effective compressed policy/constraint description size. The mechanism predicts that worst-case violation decreases sharply after repeatedly adding high-$v$ trajectories, while the certified upper bound decreases approximately as $O(\sqrt{(\mathcal{K}+\log(1/\delta))/N})$ when the policy is held fixed during certification.

## Key formulas

- $$dx_t=f(x_t,\pi_\theta(x_t,t),t;\lambda)\,dt+G\,dw_t,\qquad x_0\sim\mu_0,\quad \lambda\sim\mu_\Lambda,$$
- $$v(\tau)=\max_{r\in\{1,\ldots,R\}}\left[\frac{g_r(\tau)}{s_r}\right]_+,\qquad \tau_m^*=\arg\max_{\tau\in\mathcal{D}_m}v(\tau),$$
- $$\mathcal{C}_{m+1}=\mathcal{C}_m\cup\{\tau_m^*\},\qquad \theta_{m+1}=\arg\min_\theta\left[L_{\mathrm{task}}(\theta)+\beta\sum_{\tau\in\mathcal{C}_{m+1}}\ell_{\mathrm{safe}}(\pi_\theta,\tau)\right],$$
- $$p_v=\Pr[v(\tau)>0]\ \le\ \widehat p_v+\epsilon(N,\delta,\mathcal{K}),\qquad \widehat p_v=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{v(\tau_i)>0\},\quad \epsilon=O\!\left(\sqrt{\frac{\mathcal{K}+\log(1/\delta)}{N}}\right).$$

## Implementation notes

1. Integration point: use this as an outer loop around an RL or differentiable-simulator policy update. Keep the neural policy architecture and task loss unchanged, but add a replay buffer of simulator trajectories and a differentiable safety loss. For each constraint $g_r$, normalize by a fixed engineering scale $s_r$ so unrelated units do not dominate. 2. Pseudocode: initialize policy parameters $\theta$, constraint buffer $\mathcal{C}=\varnothing$, and repeat for $m=1,\ldots,M$: sample $B$ trajectories with independently sampled $x_0$, fixed-per-trajectory $\lambda$, and process noise; compute $v(\tau)$ for every trajectory; select the top-$q$ trajectories, or at minimum $\tau^*=\arg\max v$; append them to $\mathcal{C}$; perform $E$ gradient steps on $L_{task}(\theta)+\beta|\mathcal{C}|^{-1}\sum_{\tau\in\mathcal{C}}\operatorname{softplus}(v_\theta(\tau)/T)^2$, where $T$ is a smoothing temperature. Stop adding constraints when the batch maximum is below tolerance. 3. Computed from the paper: simulator dynamics, fixed uncertain parameters, stochastic rollouts, worst-violation selection, and the compression-style risk certificate. Estimated empirically: the policy-dependent gradients, effective description size $\mathcal{K}$, and violation probability. Use a held-out certification set generated only after training; do not reuse adaptive trajectories for $\widehat p_v$. 4. First cheap experiment: train a small MLP controller on a two-dimensional stochastic point-mass reaching task with random mass or gravity, comparing uniform-risk training, nominal covariance training, and Pick-to-Learn with equal simulator budget. Use 600 held-out rollouts and confidence $1-\delta=0.95$. The quantitative prediction is that the maximum observed normalized violation and held-out violation rate drop more rapidly per simulator rollout under Pick-to-Learn, while the certified bound follows an $N^{-1/2}$ slope when plotted against test-set size. A falsifiable target is a factor-of-two reduction in held-out violations at equal budget and agreement between the empirical rate and the upper bound within the expected confidence slack.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a self-contained Pick-to-Learn safety fine-tuning toy experiment with worst-violation selection, differentiable safety penalties, held-out certification, and fixed seeds. The mechanism manifested: observed order-statistic maxima matched B/(B+1) within 0.001, the certificate correction had log-log slope -0.500 versus the predicted -0.5, and Pick-to-Learn reduced held-out violation rate from 0.2331 to 0.0837 at equal simulator budget.

### Mechanism check

- Verdict: Built a self-contained Pick-to-Learn safety fine-tuning toy experiment with worst-violation selection, differentiable safety penalties, held-out certification, and fixed seeds. The mechanism manifested: observed order-statistic maxima matched B/(B+1) within 0.001, the certificate correction had log-log slope -0.500 versus the predicted -0.5, and Pick-to-Learn reduced held-out violation rate from 0.2331 to 0.0837 at equal simulator budget.
- Confidence: 8/10
- Limitations: This is a one-step scalar stochastic safety abstraction rather than a 2D multi-step simulator or full neural policy. It does not test RL-scale sample efficiency, Brownian dynamics, top-q selection, model misspecification, or a K-dependent certificate.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/492/bench_report.json)
- [bench_stage2.py](https://synthcore.org/code/492/bench_stage2.py)
- [experiment.py](https://synthcore.org/code/492/experiment.py)
- [report.md](https://synthcore.org/code/492/report.md)
- [report_bench_2026-08-31T155054.md](https://synthcore.org/code/492/report_bench_2026-08-31T155054.md)
- [results.json](https://synthcore.org/code/492/results.json)
- [Download all files as ZIP](https://synthcore.org/download/492)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.