✗ Failed on benchmark 2026

Poisson-Calibrated Candidate-Pool Scheduler

Implementation & benchmark of arXiv:2608.11549 — Poisson Tangent Limits and Critical Policy Switching for Sampled Bellman Operators

Usefulness8/10

Difficulty5/10

Novelty7/10

Dynamics Sample efficiency Speed Accuracy Rl Optimizer World model Probability Stochastic processes Control theory Dynamical systems

Source paper: Poisson Tangent Limits and Critical Policy Switching for Sampled Bellman Operators arXiv:2608.11549 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a fixed number of randomly sampled continuous actions with a state-dependent candidate pool whose size is chosen from the predicted extreme-value error of the best candidate. If the local action deficit has order $\|u-u^\star\|^\kappa$ in an effective dimension $d$, the best sampled action has expected Bellman error proportional to $N^{-\kappa/d}$. This gives an explicit stopping rule for increasing the pool only when the estimated residual action error is larger than the value-function or critic uncertainty.

Formulas

$$Q_V(x,u^\star)-Q_V(x,u)\approx a_x\|u-u^\star\|^{\kappa},\qquad \Delta_N(x):=Q_V(x,u^\star)-\max_{1\leq i\leq N}Q_V(x,U_i).$$

$$\Delta_N(x)=\Theta\!\left(N^{-\kappa/d_x}\right),\qquad \mathbb{E}[\Delta_N(x)]\approx C_xN^{-\kappa/d_x}.$$

$$\|\mathcal{T}V-\mathcal{T}_NV\|_\infty\leq \|\Delta_N\|_\infty,\qquad \|V^\star-V_N^\star\|_\infty\lesssim \frac{\|\Delta_N\|_\infty}{1-\beta}.$$

$$N_x=\left\lceil\left(\frac{C_x}{(1-\beta)\varepsilon_x}\right)^{d_x/\kappa}\right\rceil.$$

Mathematical statement

Assume that at state $x$, an optimal action $u^\star$ is locally isolated and $Q_V(x,u^\star)-Q_V(x,u)\approx a_x\|u-u^\star\|^\kappa$, with sampling density $q_x(u^\star)>0$ in effective dimension $d_x$. For $N$ iid candidates, the rescaled near-optimal deficits converge to a Poisson process; consequently the best-candidate deficit has scale $N^{-\kappa/d_x}$. A practical approximation uses $\mathbb{E}[\Delta_N(x)]\approx C_xN^{-\kappa/d_x}$, where $C_x$ is estimated from local candidate statistics. Because the Bellman operator is a $\beta$-contraction, a one-step action approximation error $\Delta_N$ produces a discounted value error bounded approximately by $\Delta_N/(1-\beta)$. The scheduler selects the smallest $N$ satisfying $C_xN^{-\kappa/d_x}/(1-\beta)\leq\varepsilon_x$.

Implementation notes

1. Integration point: use this in a sampled-action actor-critic or model-predictive policy layer. At each state, draw candidate actions from the actor distribution, evaluate their Q-values with the critic, and adapt the pool size before committing the action. Start with a cheap pool of N_min candidates and estimate local geometry from the top-K candidates. 2. Pseudocode: draw U[1:N], compute q_i=Q_theta(x,U_i), sort q_(1)>=q_(2)>=..., estimate a local tail exponent by fitting log(q_(1)-q_(j)) versus log(j/N), and estimate C_x from repeated minibatch states; set kappa_hat/d_hat from the fitted slope; predict e_hat=C_hat*N^(-kappa_hat/d_hat)/(1-beta); while e_hat>epsilon_x, double N, evaluate only the new candidates, and stop at N_max. Use the selected candidate for the target and optionally backpropagate through a softmax over the final pool. 3. Computed versus estimated: the contraction factor beta and candidate-pool Bellman maximization are exact; C_x, effective dimension d_x, and curvature exponent kappa are empirical estimates, preferably maintained with exponential moving averages and clipped confidence intervals. 4. First cheap experiment: use a 2D continuous-action synthetic discounted MDP with known quadratic reward and several state-dependent action dimensions. Compare fixed N in {8,32,128} against the scheduler at equal Q-evaluation budget. The predicted signature is a log-log slope of approximately -kappa/d in the measured best-action error versus N, and the adaptive pool should stop near the smallest N for which measured rollout value error is below epsilon/(1-beta). Fit the slope on held-out states; accept the mechanism only if the measured exponent is within 20 percent of the local prediction and the value error decays with the predicted power law.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built and numerically verified the Poisson-calibrated candidate-pool mechanism on a synthetic quadratic continuous-action problem. The predicted error exponents matched closely: observed slopes were -1.865 vs -2.0 for d=1 and -0.954 vs -1.0 for d=2; doubling ratios were 0.279 vs 0.25 and 0.517 vs 0.5. Scheduler scaling matched N proportional to epsilon^(-d/kappa), with slope -0.9998 vs -1.0 and an estimated asymptotic constant within 1.4% of the exact value. The noisy adaptive scheduler did not beat fixed N=128 in the heterogeneous mini-comparison, so the mechanism is validated but an efficiency win was not demonstrated.

Agent confidence: 8/10

Baseline: Fixed N=128: mean evaluations 128, mean deficit 0.01179, value-error proxy 0.11786
Idea: Adaptive scheduler: mean evaluations 117.87, mean deficit 0.01623, value-error proxy 0.16231; scaling checks passed with slopes -1.865/-0.954 and scheduler epsilon slope -0.9998

Limitations: Only a static synthetic quadratic action-selection problem was tested; no learned critic, actor distribution, multi-step RL rollout, state-dependent effective-dimension estimation, confidence intervals, or GPU implementation was evaluated. The scheduler comparison used a noisy N=8 pilot and a maximum pool of 128, likely contributing to its weaker accuracy than fixed N=128.

How to run: python3 poisson_scheduler_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.11549) Sep 2, 2026

No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 8 paired seeds).

Benchmark: Poisson_action_value
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 0.0356
Idea mean: 0.0356
Effect (Δ): 0 (−-0.0%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 1 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.0%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 15 configs

Limitations:

This is a compact supervised learned-critic control benchmark, not a multi-step actor-critic or environment rollout. Candidate selection was evaluated for signature and pool behavior but was not coupled to a downstream rollout reward; consequently the primary bench metric is critic MSE and is invariant to the inference-only pool size. The custom track is local and was not promoted into the shared read-only bench.

How to run: python3 poisson_stage2.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Built a local continuous-action control track and an end-to-end benchmark runner using the canonical bench training path, with baseline and idea sharing the same MLP critic, learning-rate search space, seeds, and training budget. The baseline and scheduler had identical test MSE (0.0356233; paired delta 0, permutation p=1.0) because candidate-pool selection does not alter critic training, and the trained-network action-regret slope was -0.494 versus the predicted -1.0, so the mechanism signature was not confirmed. The idea therefore gives no observed benchmark win.

Baseline: best_cfg lr=0.01, fixed candidate pool n=8; full 8-seed test MSE mean=0.035623283591121435, std=0.008640293958528833
Idea: best_cfg lr=0.01, epsilon=0.03; full 8-seed test MSE mean=0.035623283591121435, std=0.008640293958528833; mean scheduled pool=60.0 (sizes 32-64)
Mechanism signature: predicted trained-model candidate-regret slope=-1.0; observed mean=-0.4938379473634249; confirmed=false
Paired comparison: delta_mean=0.0, idea_wins=0/8, permutation p_value=1.0, verdict=no measurable effect

Benchmark result