✗ Failed on benchmark 2026

Bellman-Resolvent Uncertainty Targets

Implementation & benchmark of arXiv:2607.21520 — Asymptotic Analysis of Empirical Dynamic Programming in Infinite-Horizon Stochastic Optimal Control

Usefulness7/10

Difficulty6/10

Novelty6/10

Training Stability Sample efficiency Accuracy Rl Loss World model Training dynamics Probability Stochastic processes Dynamical systems Optimization

Source paper: Asymptotic Analysis of Empirical Dynamic Programming in Infinite-Horizon Stochastic Optimal Control arXiv:2607.21520 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Attach uncertainty to neural value targets by estimating the empirical one-step Bellman perturbation and propagating it through the discounted closed-loop transition operator. Use the resulting uncertainty to downweight high-variance Bellman targets or regularize the critic toward conservative predictions, especially in offline or model-based reinforcement learning.

Formulas

$$V=\mathcal{T}V,\qquad \hat{V}_{N}=\hat{\mathcal{T}}_{N}\hat{V}_{N}.$$

$$N^{1/2}(\hat{\mathcal{T}}_{N}V-\mathcal{T}V)\rightsquigarrow\mathfrak{H},$$

$$\xi=\mathfrak{H}+\gamma\mathcal{L}\xi,\qquad \xi=(I-\gamma\mathcal{L})^{-1}\mathfrak{H}=\sum_{t=0}^{\infty}\gamma^{t}\mathcal{L}^{t}\mathfrak{H}.$$

$$\operatorname{Var}[\xi]\approx (I-\gamma\hat{L})^{-1}\,\hat{\Sigma}_{H}\,(I-\gamma\hat{L})^{-\top}.$$

Mathematical statement

The paper starts from the population fixed-point equation $V=\mathcal{T}V$ and the empirical fixed-point equation $\hat{V}_{N}=\hat{\mathcal{T}}_{N}\hat{V}_{N}$, where $\mathcal{T}$ is the population dynamic-programming operator, $\hat{\mathcal{T}}_{N}$ replaces the transition or noise law by $N$ i.i.d. samples, $V$ is the population value function, and $\hat V_N$ is its empirical counterpart. The static empirical-process fluctuation is $N^{1/2}(\hat{\mathcal{T}}_{N}V-\mathcal{T}V)\rightsquigarrow\mathfrak{H}$, where $\mathfrak{H}$ is a mean-zero Gaussian process indexed by state. For a unique population optimal policy, the paper propagates this perturbation through the discounted closed-loop transition operator $\mathcal{L}$ with discount factor $\gamma\in(0,1)$. The corresponding first-order neural adaptation is $\xi=\mathfrak{H}+\gamma\mathcal{L}\xi$, hence $\xi=(I-\gamma\mathcal{L})^{-1}\mathfrak{H}=\sum_{t\ge0}\gamma^{t}\mathcal{L}^{t}\mathfrak{H}$, where $\xi$ is value uncertainty and $I$ is the identity operator. The resolvent exists because discounting makes the geometric series contractive. Estimate the covariance of $\mathfrak H$ by bootstrap resampling and propagate it with an empirical policy transition kernel.

Implementation notes

1. Integration point: modify the critic loss in fitted Q-iteration, SAC, or a small model-based RL agent. For each minibatch state $s_i$, retain the same ensemble or bootstrap transition samples used to form the Bellman target. Let $\hat H(s_i)$ be the empirical one-step target fluctuation and let $\hat\sigma_H^2(s_i)$ be its bootstrap variance. This approximates the paper's Gaussian process $\mathfrak H$. 2. Pseudocode: draw $B$ bootstrap transition batches; for each batch $b$, compute $y_i^{(b)}=r_i+\gamma Q_{\bar\phi}(s_i^{\prime(b)},\pi_{\bar\phi}(s_i^{\prime(b)}))$ and $\hat H_i^{(b)}=y_i^{(b)}-B^{-1}\sum_j y_i^{(j)}$; estimate $\hat\Sigma_H$ from the vectors $\hat H^{(b)}$. Construct a minibatch transition matrix $\hat L$ by assigning each next state to a critic-state bin or nearest prototype, or by using a learned linear feature transition $\Phi'\approx\hat L\Phi$. Solve the truncated recursion $\xi^{(0)}=0$, $\xi^{(k+1)}=\hat H+\gamma\hat L\xi^{(k)}$ for $K$ steps, and set $u_i=\sqrt{\max(\xi_i^2+\epsilon,\epsilon)}$. Train with $L=\sum_i w_i(Q_\phi(s_i,a_i)-\bar y_i)^2$, where $w_i=(u_i+\tau)^{-1}$, or use a conservative target $\bar y_i-\beta u_i$. 3. The paper supplies the scaling $N^{-1/2}$, the fixed-point propagation structure, and the discounted resolvent. Estimate $\hat H$, $\hat L$, and covariance empirically; do not attempt to build a full state-space covariance for a large neural network. Use $K\approx 1/(1-\gamma)$, diagonal covariance, and feature-space propagation for the first implementation. 4. First experiment: compare a standard DQN or SAC critic with this method on CartPole, LunarLander, and a small offline replay-buffer task, using $B=8$ bootstrap samples and $K=10$ for $\gamma=0.99$. Measure critic Bellman error, return, calibration of predicted uncertainty versus absolute TD error, and performance under reduced dataset sizes. Success is lower test Bellman error and improved return at equal environment samples; a particularly useful signal is uncertainty increasing before policy-induced distribution shift and conservative targets reducing catastrophic overestimation without slowing early loss descent.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a readable finite-state Bellman-resolvent uncertainty prototype with bootstrap one-step variance, discounted transition propagation, uncertainty-weighted fitted value iteration, and saved numerical results. The gain sweep matched 1/(1-gamma*lambda) to floating-point precision, truncation residual matched (gamma*lambda)^K/(1-gamma*lambda), the gamma*lambda>1 boundary showed explosive growth, and the propagated Gaussian variance matched within 0.014%. In the matched offline toy experiment, resolvent weighting reduced value RMSE from 1.8461 to 1.6026 (-13.2%), Bellman RMSE from 0.1285 to 0.1244 (-3.1%), and late-state RMSE from 2.2479 to 1.9511 (-13.2%); this is promising but only a small synthetic result.

Agent confidence: 9/10

Baseline: Toy mechanism: exact predicted gain/truncation formulas; offline value RMSE 1.846088, Bellman RMSE 0.1284779, late-state RMSE 2.247931
Idea: Toy mechanism: relative errors approximately 0 to 1e-13; variance relative error 0.000138; offline value RMSE 1.602595, Bellman RMSE 0.1244343, late-state RMSE 1.951077; uncertainty-error correlation 0.61067

Limitations: Not tested on CartPole, SAC/DQN, neural networks, real offline RL datasets, multiple random seeds, policy improvement, calibration metrics beyond correlation, or computational overhead. The finite-state toy uses known transition dynamics for propagation and a diagonal one-step variance approximation.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.21520) Sep 2, 2026

No significant difference vs. the tuned baseline (+11.8%, p=0.4338, wins 4 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0007
Idea mean: 0.0007
Effect (Δ): 0.0001 (+11.8%; negative = idea better)
Wins: 4 / 8 paired seeds
p-value: 0.4338 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±33.7%
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Only the registered dynamics track was tested. The experiment did not test CartPole, DQN/SAC, policy improvement, offline replay buffers, larger datasets, longer schedules, or calibration beyond the trained-model uncertainty/residual correlation. The empirical transition operator used compact nearest prototypes and a simplified supervised one-step target fluctuation.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Built and evaluated Bellman-resolvent uncertainty weighting on the structurally matched controlled-pendulum dynamics track with the shared rnn_small architecture. The baseline won slightly: mean test MSE 0.000652883 versus 0.000730047 for the idea, paired delta +0.0000771643 and permutation p=0.43375, so there is no significant improvement. The trained-model mechanism signature was also not confirmed because uncertainty/residual correlation averaged only 0.0727.

Baseline: Registered dynamics track, rnn_small, tuned sweep lr={0.001,0.003,0.01}, selected lr=0.01; 8-seed mean test MSE 0.000652883
Idea: Bellman-resolvent weighted loss at lr=0.01; 8-seed mean test MSE 0.000730047; paired delta +0.0000771643; permutation p=0.43375

Benchmark result