△ Mechanism confirmed, baseline not beaten 2026

Bellman Stopping Controller for Self-Refinement

Implementation & benchmark of arXiv:2608.10729 — Optimal Stopping of Self-Refining Foundation Models

Usefulness8/10

Difficulty5/10

Novelty5/10

Dynamics Speedup Accuracy Sample efficiency Inference speedup Optimizer Rlhf Control theory Dynamical systems Optimization Stochastic processes

Source paper: Optimal Stopping of Self-Refining Foundation Models arXiv:2608.10729 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Attach a value-based stopping controller to any verifier-guided refinement loop. After each generated answer and verifier evaluation, estimate the value of accepting the current output and the value of continuing for one or more additional refinements; stop when the expected gain from continuation is no larger than its compute cost. The controller learns a score-dependent stopping boundary instead of using a fixed iteration count.

Formulas

$$x_{k+1}=f(x_k,w_k),\qquad w_k\overset{\mathrm{i.i.d.}}{\sim}P_w,\qquad k=0,\ldots,N-1.$$

$$V_N(x)=g(x),\qquad V_k(x)=\max\left\{g(x),-c+\mathbb{E}_{w\sim P_w}\left[V_{k+1}\left(f(x,w)\right)\right]\right\}.$$

$$\mu_k(x)=\begin{cases}\mathsf{S},&g(x)\geq -c+\mathbb{E}[V_{k+1}(f(x,w))],\\\mathsf{C},&g(x)<-c+\mathbb{E}[V_{k+1}(f(x,w))].\end{cases}$$

$$\Delta_k(x)=\mathbb{E}[V_{k+1}(f(x,w))]-g(x);\qquad \text{continue iff }\Delta_k(x)>c.$$

Mathematical statement

Let $x_k$ be the observable refinement state at iteration $k$, including verifier score, uncertainty, error categories, and optionally an answer embedding; let $w_k\sim P_w$ be random refinement feedback; let $x_{k+1}=f(x_k,w_k)$ be the stochastic state transition; let $g(x)\geq 0$ be terminal quality payoff; let $c>0$ be the cost of one additional refinement; and let $N$ be the maximum number of refinements. The finite-horizon value is $V_N(x)=g(x)$, and for $k<N$, $V_k(x)=\max\{g(x),-c+\mathbb{E}_{w\sim P_w}[V_{k+1}(f(x,w))]\}$. Stop when the immediate payoff is at least the continuation value. Continue only when the expected future value exceeds the current payoff by more than $c$.

Implementation notes

1. Exact integration point: place the controller after every model refinement and verifier call. The base model receives the prompt, current answer, and verifier feedback, produces a candidate refinement, and exposes a state vector $x_k$ containing verifier score, score uncertainty, detected-error count and types, token count, elapsed cost, and optionally a frozen answer embedding. Define $g(x_k)$ as calibrated probability of passing the final evaluator or as normalized task utility.

2. Pseudocode: initialize $x=x_0$. For $k=0,\ldots,N-1$, compute $g=g(x)$, sample $M$ possible feedback or refinement outcomes $w^{(m)}$, predict $V_{k+1}(f(x,w^{(m)}))$, and set $\widehat Q_C=-c+M^{-1}\sum_m V_{k+1}(f(x,w^{(m)}))$. If $g\geq\widehat Q_C$, return the current answer; otherwise perform one refinement, obtain the actual verifier result, update $x$, and continue. At $k=N$, return unconditionally. A cheaper implementation learns $Q_C(x,k)$ with a small MLP and uses target $y=-c+V_{k+1}(x_{k+1})$, with $V_k(x)=\max(g(x),Q_C(x,k))$.

3. Computed versus estimated: the Bellman maximum and stopping decision are exact given $g$, $c$, and a continuation-value estimate. The transition law, verifier calibration, and continuation values are estimated from logged trajectories or Monte Carlo rollouts. Use stochastic approximation with step size $\alpha_t$: $Q_C\leftarrow Q_C+\alpha_t(y-Q_C)$. Measure costs in model calls, generated tokens, or wall-clock time.

4. First cheap experiment: use a small code model on HumanEval or MBPP with a verifier, permit $N=5$ refinements, and compare fixed stopping at 1, 3, and 5 iterations against the adaptive controller. Sweep $c$ over token-equivalent costs. The quantitative prediction is that continuation frequency decreases monotonically with $c$, and observed continuation should occur primarily when estimated $\Delta_k(x)>c$. Plot stopping probability against verifier score; for a mostly one-dimensional score state it should show an approximately monotone threshold. At equal pass rate, adaptive stopping should use fewer refinement calls than fixed-budget inference.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built an exact finite-horizon Bellman stopping controller for a stochastic score-refinement environment, with Monte Carlo comparison against fixed 1/3/5-step budgets. The mechanism manifested: continuation decreased monotonically with cost, the stop region was an upper score threshold that moved from 0.60 at cost 0.02 to 0.10 at cost 0.04, and all Bellman residuals were 0. At cost 0.038, adaptive stopping used 3.626 calls versus 5 fixed calls while achieving objective 0.00947 versus 0.00690 for fixed-5; this is a promising toy signal, not evidence of real-model improvement.

Agent confidence: 9/10

Baseline: At cost 0.038: fixed-1 = 1.000 calls, quality 0.04245, objective 0.00445; fixed-3 = 3.000 calls, quality 0.12281, objective 0.00881; fixed-5 = 5.000 calls, quality 0.19690, objective 0.00690.
Idea: At cost 0.038: adaptive = 3.626 calls, quality 0.14726, objective 0.00947. Exact Bellman residual = 0.0 and Delta>cost decisions matched the policy for every tested state. Cost sweep calls: 5.00 at 0.02, 4.60 at 0.035, 3.62 at 0.038, 2.40 at 0.04, and 0 at 0.045.

Limitations: Only a synthetic one-dimensional score environment with known transitions and payoff was tested. The continuation value was computed exactly rather than learned from logged verifier/model trajectories; no LLM, HumanEval/MBPP, calibrated verifier, embedding state, wall-clock measurement, or GPU experiment was tested.

How to run: python3 bellman_stopping_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.10729) Sep 2, 2026

Significantly WORSE than the tuned baseline (+69.4%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0021
Idea mean: 0.0035
Effect (Δ): 0.0014 (+69.4%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±15.0%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 9 configs

Limitations:

The controller uses a lightweight self-consistency verifier proxy rather than an LLM, external verifier, or learned continuation-value MLP. Refinement costs are normalized proxy costs; wall-clock and token costs were not measured.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

Implemented and ran the Bellman stopping controller on the registered dynamics track using paired rnn_small systems and 8 seeds. The Bellman recursion had zero residual, and trained-model continuation frequency decreased monotonically with cost, but the tuned adaptive system significantly worsened test MSE versus the tuned fixed-stop baseline; therefore the idea does not win this benchmark.

Baseline: best_cfg={lr:0.006, steps:1}; full mean test MSE=0.0020863086683675647, std=0.00045500750701757853
Comparison: paired delta_mean=0.0014472102338913828, p_value=0.0081, idea_wins=0/8, verdict=idea worse (significant)
Idea: best_cfg={lr:0.006, cost:0.003}; full mean test MSE=0.0035335189022589475, std=0.0006978225321427611

Benchmark result