✗ Failed on benchmark 2026

Level-Adaptive Replay Memory

Implementation & benchmark of arXiv:2607.22211 — Closed-Loop Generative Selection: Convergence, Memory, and Noisy Oracles

Usefulness7/10

Difficulty5/10

Novelty6/10

Training Sample efficiency Stability Speedup Diffusion Vae Rl Training Probability Stochastic processes Optimization

Source paper: Closed-Loop Generative Selection: Convergence, Memory, and Noisy Oracles arXiv:2607.22211 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use the recent history of generator outputs as a controllable training window instead of fixing the replay-memory depth globally. Estimate how quickly each fitness level improves as more same-level examples enter the window, and increase memory only when the measured escape probability improves enough to justify the extra stale data.

Formulas

$$\mathcal{W}_{k}(j)=\bigl\{S=(Q_{1},\dots,Q_{m})\in\Omega_{N}^{m}:\ell(Q_{i})=k\ (m-j<i\leq m),\quad\ell(Q_{i})\leq k\ (i\leq m-j)\bigr\}$$

$$\rho_{k}(j)=\inf_{S\in\mathcal{W}_{k}(j)}G(S)\bigl(A_{k}^{+}\mid Q_{m}\bigr),\qquad r_{k}(j)=1-\bigl(1-\rho_{k}(j)\bigr)^{M}$$

$$\widehat{\rho}_{k,m}=\frac{\#\{\text{proposals at level }k\text{ that reach the next level using memory }m\}}{\#\{\text{proposals evaluated at level }k\}}$$

$$m_{t+1}=\begin{cases}m_t+1,&\widehat r_{k,m_t+1}-\widehat r_{k,m_t}>\tau\\m_t-1,&\widehat r_{k,m_t}-\widehat r_{k,m_t-1}< -\tau\\m_t,&\text{otherwise,}\end{cases}\qquad \widehat r_{k,m}=1-(1-\widehat\rho_{k,m})^M$$

Mathematical statement

The paper defines a window of depth m as S=(Q_1,...,Q_m), where Q_i is the elite population stored at round i, and the current fitness level is denoted k. The level-k profile is W_k(j)={S: ell(Q_i)=k for the j most recent pools and ell(Q_i)<=k for older pools}, where ell(Q_i) is the level of pool Q_i. For a conditional generator G(S)(·|Q_m), rho_k(j)=inf_{S in W_k(j)} G(S)(A_k^+|Q_m) is the worst-case probability that one proposal enters the next level A_k^+; with M independent proposals, r_k(j)=1-(1-rho_k(j))^M is the probability of escaping level k in one round. Under monotone learning, increasing memory improves rho_k, but without monotone learning the expected sojourn can be U-shaped in m. The adaptation estimates rho_hat_{k,m} empirically and chooses the smallest memory depth whose escape-rate gain exceeds a tolerance.

Implementation notes

Integrate this at the dataset-construction step of an iterative generator: diffusion fine-tuning, VAE molecular generation, or policy optimization. At round t, retain the last m_t elite batches Q_{t-m_t+1:t}, concatenate them with optional recency weights, and fine-tune the generator on this window. Track a discrete fitness level k using quantile bins or fixed reward thresholds; a proposal is an escape when its oracle score crosses the next threshold. Initialize m=1 and a bounded range [1,m_max]. For each round, sample M candidates from G_theta conditioned on the current elite pool, score them, and estimate rho_hat[k,m] from the fraction crossing the next level. Store separate counters for each pair (k,m), preferably with exponential decay so estimates adapt when the generator changes. Every K rounds, probe one neighboring memory depth for one validation round; compute r_hat[k,m]=1-(1-rho_hat[k,m])^M, and update m using the threshold rule with a confidence margin tau. Train theta on the selected window with the same optimizer and gradient-step budget as the baseline. The paper supplies the level-profile and M-fold escape formulas; rho_hat, quantile levels, and tau are empirical choices. First test a small diffusion or VAE generator on a molecular benchmark, comparing fixed windows m in {1,2,4,8,16} with the adaptive controller under equal oracle-call and gradient budgets. Success means fewer oracle evaluations to reach a fixed top-quantile reward and avoidance of large-window degradation under stale or distribution-shifted elite data.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a controlled level-adaptive replay-memory simulator with empirical rho estimation, M-fold escape-rate calculation, neighbor probing, and fixed-window baselines. The core mechanism manifested: the exact escape formula matched Monte Carlo within 0.0017, monotone memory improvement held for 100% of tested increments, and increasing stale-data penalty shifted the optimal depth from m=16 to m=3-5, with observed optima within one depth of prediction. The adaptive controller did not win the matched toy comparison (65 escapes versus 73 for fixed m=2), so the evidence supports the mechanism but not a performance improvement.

Agent confidence: 8/10

Baseline: Fixed windows over 80 rounds: m=1: 60 escapes, m=2: 73, m=4: 69, m=8: 43, m=16: 0; best fixed m=2.
Idea: Adaptive controller: 65/80 escapes, mean depth 3.75, final depth 6. Formula Monte Carlo absolute error <=0.0017; monotone improvement fraction 1.0; stale-penalty predicted/observed optima: 0.0: 16/15, 0.005: 8/8, 0.015: 5/5, 0.03: 3/4, 0.05: 3/3, 0.08: 3/3. Low-rho r/(M*rho) ranged 0.985-1.000 for M=1-16.

Limitations: This is a synthetic Bernoulli level-escape simulator, not a neural generator, diffusion/VAE, molecular benchmark, or policy-optimization task. It does not test wall-clock speed, gradient/FLOP budgets, distributional level profiles, confidence intervals for online adaptation, or robustness across many random seeds; the controller uses a simple neighbor-probing policy and fixed toy stale-data curve.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.22211) Aug 31, 2026

Significantly WORSE than the tuned baseline (+215.2%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.3346
Idea mean: 23.12
Effect (Δ): 15.7854 (+215.2%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±60.0%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 12 configs

Limitations:

Only the registered tabular track was tested. The implementation uses target-quantile escape proxies and replay pools rather than a generator-conditioned molecular or diffusion system; wall-clock speed, oracle-call accounting, confidence intervals for online estimates, and the other built-in tracks were not tested.

How to run: python3 level_adaptive_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

Level-adaptive replay memory was implemented and evaluated on the registered tabular Friedman regression track with the same mlp_tiny architecture and equal training budget. The adaptive system was significantly worse than the tuned fixed-window baseline: MSE 23.119989 versus 7.334557, paired delta +15.785432, permutation p=0.0081, and 0/8 paired wins. The trained-model escape signature was confirmed (predicted 0.440159 versus observed 0.440000), but the mechanism did not yield a task-metric improvement.

Baseline: Fixed replay depth 8, lr 0.01; 8-seed test MSE mean 7.334557235240936, std 0.824177040857451.
Idea: Adaptive replay, selected lr 0.01; 8-seed test MSE mean 23.1199893951416, std 5.121285566952831; paired delta +15.785432159900665; p=0.0081; 0/8 wins.

Benchmark result