✗ Failed on benchmark 2026

Tangential Bellman Tie Resolver

Implementation & benchmark of arXiv:2608.11549 — Poisson Tangent Limits and Critical Policy Switching for Sampled Bellman Operators

Usefulness8/10

Difficulty7/10

Novelty8/10

Dynamics Stability Accuracy Sample efficiency Rl World model Optimizer Regularization Control theory Dynamical systems Probability Stochastic processes Linear algebra

Source paper: Poisson Tangent Limits and Critical Policy Switching for Sampled Bellman Operators arXiv:2608.11549 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

When several action branches have nearly equal Q-values, select among them using their long-horizon transition consequences rather than only noisy one-step critic values. Construct a finite sampled approximation to the paper's marked tangential Bellman operator: each candidate receives a local deficit mark and a continuation-value mark, and the branch scores are iterated through a discounted fixed point. Under a perturbation of size comparable to the finite-pool extreme-value gap, the resulting policy should switch continuously instead of exhibiting unstable arbitrary winner-take-all decisions.

Formulas

$$(\mathcal{H}z)_b(x)=\delta_b(x)+\beta\sum_yP_b(y\mid x)\min_{c\leq B}\{z_c(y)+\delta_c(y)\},\qquad z^\star=\mathcal{H}z^\star.$$

$$\|\mathcal{H}z-\mathcal{H}w\|_\infty\leq\beta\|z-w\|_\infty,\qquad \|z^{(k)}-z^\star\|_\infty\leq\beta^k\|z^{(0)}-z^\star\|_\infty.$$

$$s_b(x)=z_b^\star(x)+\delta_b(x),\qquad \pi_b(x)=\frac{\exp[-s_b(x)/\tau]}{\sum_c\exp[-s_c(x)/\tau]}.$$

$$\text{critical coordinate:}\quad \xi=\tau_{\mathrm{tie}}N^{\kappa/d},\qquad \pi_b\approx F_b(\xi).$$

Mathematical statement

Let branches $b\in\{1,\ldots,B\}$ represent neighborhoods of tied or nearly tied actions. For branch $b$, let $\delta_b\geq0$ be its local immediate Q deficit relative to the best branch and let $P_b(y\mid x)$ be its transition kernel or learned ensemble transition distribution. Define the tangential Bellman map $(\mathcal{H}z)_b(x)=\delta_b(x)+\beta\sum_yP_b(y\mid x)\min_c[z_c(y)+\delta_c(y)]$. For $\beta<1$, $\mathcal{H}$ is a sup-norm contraction and has a unique fixed point $z^\star$. Use $s_b=z_b^\star+\delta_b$ as branch scores. A critical switching layer occurs when the branch perturbation $\tau$ is of the same order as the sampled extreme-value gap $N^{-\kappa/d}$, so the normalized selection probability should be a smooth function of $\tau N^{\kappa/d}$, rather than a discontinuous function of $\tau$.

Implementation notes

1. Integration point: add this as a tie-resolution head on top of a continuous-action critic. Partition candidate actions into B local clusters using action-space distance or a learned mixture policy. For each cluster, estimate the immediate deficit and continuation transition distribution; use the resulting branch score during action selection and target construction. 2. Pseudocode: cluster candidate actions; for each branch b compute delta[b]=q_max-q_branch and obtain P_b either from a learned dynamics ensemble or by averaging next-state samples generated after executing representative actions; initialize z=0; repeat K times: for every state y compute m[y]=min_c(z[c,y]+delta[c,y]); update z_new[b,x]=delta[b,x]+beta*mean_y~P_b[m[y]]; stop when max_abs(z_new-z)<tol; form s[b]=z[b]+delta[b]; select with softmax(-s/tau), or use the branch's best candidate. 3. Computed versus estimated: contraction and iteration error are known analytically; delta, P_b, and the number of competing branches are estimated from the critic, replay data, and model ensemble. Use a soft-min during early training and anneal its temperature only after the fixed-point residual is small. 4. First cheap experiment: create a discounted gridworld or finite-state MDP with two action branches having identical immediate reward but different transition kernels, then add a controllable reward perturbation tau_tie and Gaussian critic noise. Compare ordinary argmax, random tie-breaking, and the tangential resolver in SAC-style sampled action selection. Sweep N and perturbation magnitude. The predicted signature is that the branch-selection curve collapses when plotted against tau_tie*N^(kappa/d), with a smooth transition near order one; ordinary argmax should switch at a noise-dependent location. Also verify geometric convergence of the inner solver with residual ratio beta, and reject the method if the observed ratio is not bounded by approximately beta plus estimation noise.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a finite-state Tangential Bellman Tie Resolver with value iteration, softmax branch selection, contraction checks, and reproducible scaling sweeps. The mechanism manifested: fixed-point error ratios matched beta across beta={0.2,0.5,0.8,0.95} to within about 5e-7, and resolver transition widths scaled exactly as 1/N, with width*N approximately 0.769 and normalized-curve deviation 6.4e-14. This is a strong toy-level confirmation, though not evidence of an RL performance win.

Agent confidence: 8/10

Baseline: Noisy hard argmin transition width = 0.1331 for every N; width*N = {2.662, 6.655, 13.310, 26.620}. Under perturbation 0.03 and Gaussian noise sigma=0.06, best-branch selection frequency was 0.2077.
Idea: Tangential resolver transition widths for N={20,50,100,200} = {0.03846, 0.01538, 0.00769, 0.00385}; width*N = {0.76914, 0.76914, 0.76914, 0.76914}. Normalized curve max deviation was 6.35e-14. Soft resolver probability of the best branch under the direct noisy test was 0.2256.

Limitations: Only a synthetic finite-state MDP was tested; no continuous-action critic, learned dynamics ensemble, SAC integration, replay data, environment benchmark, compute/FLOP comparison, or statistically repeated RL training was performed. The N-scaling experiment uses a controlled temperature rule tau=0.35/N and therefore verifies predicted mathematical scaling rather than estimating an empirical extreme-value exponent from sampled actions.

How to run: python3 tie_resolver_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.11549) Sep 1, 2026

Significantly WORSE than the tuned baseline (+172833.3%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0005
Idea mean: 0.7882
Effect (Δ): 0.7877 (+172833.3%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±8219.0%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 6 configs

Limitations:

Only the built-in dynamics track was tested, with 1,200 training and 400 test examples and 10 epochs rather than the harness defaults. The implementation is an auxiliary supervised counterfactual-branch loss rather than a complete continuous-action actor-critic or SAC integration with learned transition ensembles; no environment interaction, replay efficiency, FLOP, or larger-scale RL evaluation was tested.

How to run: python3 bench_stage2.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

Built a Tangential Bellman Tie Resolver auxiliary training intervention on the structurally matched dynamics/rnn_small pendulum benchmark. The fixed-point iteration numerically contracted at the predicted beta bound (late ratio 0.729995 versus beta 0.73), but the idea scored substantially worse than the tuned baseline: MSE 0.788188 versus 0.0004558, with paired delta +0.787733 and permutation p=0.0081. Trained-model resolver weights had a very small mean branch span (0.00367), so the predicted continuous switching behavior was not confirmed; the system therefore did not work on this benchmark.

Baseline: dynamics/rnn_small; 8-seed test MSE mean=0.0004557761, std=0.0000904587; best sweep cfg lr=0.005, weight_decay=0.0; sweep covered lr={0.001,0.003,0.005} and weight_decay={0,0.0001}
Idea: dynamics/rnn_small; 8-seed test MSE mean=0.78818847, std=0.04192548; best idea cfg lr=0.003, weight_decay=0.0; paired delta (idea-baseline)=+0.78773270, p=0.0081, idea wins=0/8
Mechanism signature: Contraction ratio late=0.7299954 for beta=0.73; trained-model resolver weight span mean=0.0036721; hard-best frequency mean=0.4100; confirmed=false

Benchmark result