# Bellman Stopping Controller for Self-Refinement

- ID: 2092
- Canonical URL: https://synthcore.org/idea/2092/bellman-stopping-controller-for-self-refinement
- API JSON: https://synthcore.org/api/idea/2092.json
- API Markdown: https://synthcore.org/api/idea/2092.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.10729](https://arxiv.org/abs/2608.10729)
- Category: dynamics
- Solves: speedup, accuracy, sample-efficiency
- ML areas: inference-speedup, optimizer, rlhf
- Math tags: control-theory, dynamical-systems, optimization, stochastic-processes
- Ratings: usefulness 8/10; difficulty 5/10; novelty 5/10

## Idea description

Attach a value-based stopping controller to any verifier-guided refinement loop. After each generated answer and verifier evaluation, estimate the value of accepting the current output and the value of continuing for one or more additional refinements; stop when the expected gain from continuation is no larger than its compute cost. The controller learns a score-dependent stopping boundary instead of using a fixed iteration count.

## Mathematical statement

Let \(x_k\) be the observable refinement state at iteration \(k\), including verifier score, uncertainty, error categories, and optionally an answer embedding; let \(w_k\sim P_w\) be random refinement feedback; let \(x_{k+1}=f(x_k,w_k)\) be the stochastic state transition; let \(g(x)\geq 0\) be terminal quality payoff; let \(c>0\) be the cost of one additional refinement; and let \(N\) be the maximum number of refinements. The finite-horizon value is \(V_N(x)=g(x)\), and for \(k<N\), \(V_k(x)=\max\{g(x),-c+\mathbb{E}_{w\sim P_w}[V_{k+1}(f(x,w))]\}\). Stop when the immediate payoff is at least the continuation value. Continue only when the expected future value exceeds the current payoff by more than \(c\).

## Key formulas

- $$x_{k+1}=f(x_k,w_k),\qquad w_k\overset{\mathrm{i.i.d.}}{\sim}P_w,\qquad k=0,\ldots,N-1.$$
- $$V_N(x)=g(x),\qquad V_k(x)=\max\left\{g(x),-c+\mathbb{E}_{w\sim P_w}\left[V_{k+1}\left(f(x,w)\right)\right]\right\}.$$
- $$\mu_k(x)=\begin{cases}\mathsf{S},&g(x)\geq -c+\mathbb{E}[V_{k+1}(f(x,w))],\\\mathsf{C},&g(x)<-c+\mathbb{E}[V_{k+1}(f(x,w))].\end{cases}$$
- $$\Delta_k(x)=\mathbb{E}[V_{k+1}(f(x,w))]-g(x);\qquad \text{continue iff }\Delta_k(x)>c.$$

## Implementation notes

1. Exact integration point: place the controller after every model refinement and verifier call. The base model receives the prompt, current answer, and verifier feedback, produces a candidate refinement, and exposes a state vector \(x_k\) containing verifier score, score uncertainty, detected-error count and types, token count, elapsed cost, and optionally a frozen answer embedding. Define \(g(x_k)\) as calibrated probability of passing the final evaluator or as normalized task utility.

2. Pseudocode: initialize \(x=x_0\). For \(k=0,\ldots,N-1\), compute \(g=g(x)\), sample \(M\) possible feedback or refinement outcomes \(w^{(m)}\), predict \(V_{k+1}(f(x,w^{(m)}))\), and set \(\widehat Q_C=-c+M^{-1}\sum_m V_{k+1}(f(x,w^{(m)}))\). If \(g\geq\widehat Q_C\), return the current answer; otherwise perform one refinement, obtain the actual verifier result, update \(x\), and continue. At \(k=N\), return unconditionally. A cheaper implementation learns \(Q_C(x,k)\) with a small MLP and uses target \(y=-c+V_{k+1}(x_{k+1})\), with \(V_k(x)=\max(g(x),Q_C(x,k))\).

3. Computed versus estimated: the Bellman maximum and stopping decision are exact given \(g\), \(c\), and a continuation-value estimate. The transition law, verifier calibration, and continuation values are estimated from logged trajectories or Monte Carlo rollouts. Use stochastic approximation with step size \(\alpha_t\): \(Q_C\leftarrow Q_C+\alpha_t(y-Q_C)\). Measure costs in model calls, generated tokens, or wall-clock time.

4. First cheap experiment: use a small code model on HumanEval or MBPP with a verifier, permit \(N=5\) refinements, and compare fixed stopping at 1, 3, and 5 iterations against the adaptive controller. Sweep \(c\) over token-equivalent costs. The quantitative prediction is that continuation frequency decreases monotonically with \(c\), and observed continuation should occur primarily when estimated \(\Delta_k(x)>c\). Plot stopping probability against verifier score; for a mostly one-dimensional score state it should show an approximately monotone threshold. At equal pass rate, adaptive stopping should use fewer refinement calls than fixed-budget inference.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built an exact finite-horizon Bellman stopping controller for a stochastic score-refinement environment, with Monte Carlo comparison against fixed 1/3/5-step budgets. The mechanism manifested: continuation decreased monotonically with cost, the stop region was an upper score threshold that moved from 0.60 at cost 0.02 to 0.10 at cost 0.04, and all Bellman residuals were 0. At cost 0.038, adaptive stopping used 3.626 calls versus 5 fixed calls while achieving objective 0.00947 versus 0.00690 for fixed-5; this is a promising toy signal, not evidence of real-model improvement.

### Mechanism check

- Verdict: Built an exact finite-horizon Bellman stopping controller for a stochastic score-refinement environment, with Monte Carlo comparison against fixed 1/3/5-step budgets. The mechanism manifested: continuation decreased monotonically with cost, the stop region was an upper score threshold that moved from 0.60 at cost 0.02 to 0.10 at cost 0.04, and all Bellman residuals were 0. At cost 0.038, adaptive stopping used 3.626 calls versus 5 fixed calls while achieving objective 0.00947 versus 0.00690 for fixed-5; this is a promising toy signal, not evidence of real-model improvement.
- Confidence: 9/10
- Limitations: Only a synthetic one-dimensional score environment with known transitions and payoff was tested. The continuation value was computed exactly rather than learned from logged verifier/model trajectories; no LLM, HumanEval/MBPP, calibrated verifier, embedding state, wall-clock measurement, or GPU experiment was tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bellman_stopping_experiment.py](https://synthcore.org/code/780/bellman_stopping_experiment.py)
- [bench_report.json](https://synthcore.org/code/780/bench_report.json)
- [report.md](https://synthcore.org/code/780/report.md)
- [report_bench_2026-09-01T102456.md](https://synthcore.org/code/780/report_bench_2026-09-01T102456.md)
- [results.json](https://synthcore.org/code/780/results.json)
- [stage2_bench.py](https://synthcore.org/code/780/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/780)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.