✗ Failed on benchmark 2026

Confidence-Sequential Noisy Selection

Implementation & benchmark of arXiv:2607.22211 — Closed-Loop Generative Selection: Convergence, Memory, and Noisy Oracles

Usefulness8/10

Difficulty4/10

Novelty5/10

Training Sample efficiency Stability Accuracy Rl Diffusion Vae Training Data augmentation Statistics Probability Information theory Optimization

Source paper: Closed-Loop Generative Selection: Convergence, Memory, and Noisy Oracles arXiv:2607.22211 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

When candidate rewards are noisy, do not score every candidate once and immediately retrain on the apparent elites. Sequentially resample only candidates near the selection boundary and certify their ordering with a mean, median-of-means, trimmed mean, or sign test selected according to the observed tail behavior.

Formulas

$$r\ \gtrsim\ \sigma^{2}\gamma^{-2}\ln(2/\delta)\quad\text{(sub-Gaussian sample mean or finite-variance median-of-means)}$$

$$r\ \gtrsim\ \bigl(\nu\,\gamma^{-(1+\alpha)}\bigr)^{1/\alpha}\ln(2/\delta),\qquad 0<\alpha<1$$

$$r\ \gtrsim\ \frac{\ln(2/\delta)}{2\,(p_{\gamma}-\tfrac{1}{2})^{2}},\qquad p_{\gamma}=\mathbb{P}(\xi<\gamma)$$

$$\mathbb{E}[\mathcal{E}_{\varepsilon}]\ \lesssim\ \sum_{k}\frac{r_k}{q_k}\ \approx\ 2\sigma^{2}\ln\frac{2}{\delta}\sum_{k}\frac{1}{\Delta_k^{2}q_k}$$

Mathematical statement

Let the observed score of candidate x be Y=phi(x)+xi, where phi(x) is its true fitness and xi is zero-mean noise. To certify a true margin gamma>0 with error at most delta, the paper gives r proportional to sigma^2 gamma^{-2} log(2/delta) for sub-Gaussian sample means, the same rate with a larger constant for median-of-means under finite variance sigma^2, r proportional to (nu gamma^{-(1+alpha)})^{1/alpha} log(2/delta) under a finite (1+alpha)-moment nu with 0<alpha<1, and r proportional to log(2/delta)/(2(p_gamma-1/2)^2) for a sign test without moment assumptions, where p_gamma=P(xi<gamma). The paper's noisy-oracle cost multiplies the noiseless per-level escape cost 1/q_k by certification repetitions r_k, giving approximately 2 sigma^2 log(2/delta)/(Delta_k^2 q_k), where q_k is the probability of escaping level k and Delta_k is the fitness margin that must be resolved. The transferable principle is to spend repetitions adaptively only when Delta is small or ranking confidence is low.

Implementation notes

Integrate this into the candidate-ranking operation before an elite batch is added to a generator-training dataset or an RL replay buffer. First evaluate every candidate once. Identify candidates whose confidence intervals overlap the current elite cutoff, and repeatedly evaluate only those candidates; candidates far above or below the cutoff need no additional oracle calls. For light-tailed rewards, use median-of-means: with total repetition count r, divide observations into b=ceil(8 log(2/delta)) blocks, average within each block, and use the median of block means. For heavy-tailed rewards, use a trimmed mean if an estimated moment order is available, or a pairwise sign test when no reliable scale estimate exists. Maintain per-candidate samples; compute a robust score and confidence radius; stop sampling candidate i when its lower confidence bound exceeds the cutoff's upper confidence bound or vice versa; otherwise allocate another batch of repetitions. Set gamma to the required reward-resolution margin and delta to 0.01 divided by the number of candidates and rounds. Estimate sigma or nu from pilot samples only for scheduling; the sign test does not require a moment estimate. The paper determines the repetition-rate scaling and level-wise cost inflation, while the confidence intervals and cutoff are implementation choices. Test on a small generator or PPO policy with Gaussian, Student-t, and Cauchy synthetic reward noise. Compare one-shot scoring, repeated sample means, and adaptive median-of-means/sign selection. Success is higher true-elite precision at the same oracle budget, fewer calls to reach a target reward, and stable training under heavy tails.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a sequential top-k noisy selector that scores every candidate once and resamples only candidates whose confidence intervals cross the elite boundary. The math checks passed: Gaussian certification error matched the exact finite-sample curve within 11.4% relative error, inverse-square repetition scaling held with gamma²r approximately 8–9 for practical margins, and the certification bound stayed below the 0.05 target. Adaptive selection improved precision over fixed 20-sample scoring for Gaussian (0.788 vs 0.650) and Student-t noise (0.718 vs 0.541), but used substantially more calls; under Cauchy noise the mean-based implementation gave only a weak precision improvement (0.247 vs 0.228).

Agent confidence: 8/10

Baseline: Fixed 20 evaluations per candidate: Gaussian precision 0.650 at 480 calls; Student-t 0.541 at 480 calls; Cauchy 0.228 at 480 calls.
Idea: Adaptive confidence selection: Gaussian precision 0.788 at 1995 calls; Student-t 0.718 at 1981 calls; Cauchy 0.247 at 616 calls. Math checks: bound pass, exact Gaussian error-curve pass, inverse-square scaling pass.

Limitations: The adaptive method was not compared at a strictly equal oracle budget, and its confidence radius assumes known Gaussian-scale sigma even for heavy-tailed noise. The Cauchy path used a sample mean rather than the planned sign-test or trimmed-mean estimator, no neural-network or RL training loop was tested, and the empirical sweeps were small synthetic experiments.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.22211) Aug 30, 2026

Ahead of the baseline (−0.4%) but not statistically significant (p=0.6756, wins 5 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 9.9869
Idea mean: 9.9513
Effect (Δ): -0.0356 (−0.4%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.6756 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the matched tabular track was tested; vision, sequence, and dynamics were not run. The implementation used synthetic noisy residual utilities within the tabular training loop rather than a full RL replay buffer or generator-training application. The CPU fallback path was not separately benchmarked.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented confidence-sequential noisy hard-example selection as a training-loop intervention on the matched tabular Friedman regression track, using the same mlp_tiny architecture and paired eight-seed protocol. The idea reached test MSE 9.951270 versus tuned baseline 9.986904, but the paired permutation test was non-significant (delta -0.035634, p=0.6756), so there is no demonstrated benchmark win. The NN-scale call prediction was not quantitatively confirmed, and the finite-sample math check failed its criteria.

Baseline: Tuned baseline lr=0.006; full 8-seed test MSE mean 9.986904, std 1.189758.
Idea: Best idea lr=0.006 from the parity-preserving three-value sweep; full 8-seed test MSE mean 9.951270, std 1.103466.
Idea wins: 5
N pairs: 8
Paired delta: -0.035634 (lower is better)
Permutation p value: 0.6756

Benchmark result