# Tangential Bellman Tie Resolver

- ID: 2137
- Canonical URL: https://synthcore.org/idea/2137/tangential-bellman-tie-resolver
- API JSON: https://synthcore.org/api/idea/2137.json
- API Markdown: https://synthcore.org/api/idea/2137.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.11549](https://arxiv.org/abs/2608.11549)
- Category: dynamics
- Solves: stability, accuracy, sample-efficiency
- ML areas: rl, world-model, optimizer, regularization
- Math tags: control-theory, dynamical-systems, probability, stochastic-processes, linear-algebra
- Ratings: usefulness 8/10; difficulty 7/10; novelty 8/10

## Idea description

When several action branches have nearly equal Q-values, select among them using their long-horizon transition consequences rather than only noisy one-step critic values. Construct a finite sampled approximation to the paper's marked tangential Bellman operator: each candidate receives a local deficit mark and a continuation-value mark, and the branch scores are iterated through a discounted fixed point. Under a perturbation of size comparable to the finite-pool extreme-value gap, the resulting policy should switch continuously instead of exhibiting unstable arbitrary winner-take-all decisions.

## Mathematical statement

Let branches \(b\in\{1,\ldots,B\}\) represent neighborhoods of tied or nearly tied actions. For branch \(b\), let \(\delta_b\geq0\) be its local immediate Q deficit relative to the best branch and let \(P_b(y\mid x)\) be its transition kernel or learned ensemble transition distribution. Define the tangential Bellman map \((\mathcal{H}z)_b(x)=\delta_b(x)+\beta\sum_yP_b(y\mid x)\min_c[z_c(y)+\delta_c(y)]\). For \(\beta<1\), \(\mathcal{H}\) is a sup-norm contraction and has a unique fixed point \(z^\star\). Use \(s_b=z_b^\star+\delta_b\) as branch scores. A critical switching layer occurs when the branch perturbation \(\tau\) is of the same order as the sampled extreme-value gap \(N^{-\kappa/d}\), so the normalized selection probability should be a smooth function of \(\tau N^{\kappa/d}\), rather than a discontinuous function of \(\tau\).

## Key formulas

- $$(\mathcal{H}z)_b(x)=\delta_b(x)+\beta\sum_yP_b(y\mid x)\min_{c\leq B}\{z_c(y)+\delta_c(y)\},\qquad z^\star=\mathcal{H}z^\star.$$
- $$\|\mathcal{H}z-\mathcal{H}w\|_\infty\leq\beta\|z-w\|_\infty,\qquad \|z^{(k)}-z^\star\|_\infty\leq\beta^k\|z^{(0)}-z^\star\|_\infty.$$
- $$s_b(x)=z_b^\star(x)+\delta_b(x),\qquad \pi_b(x)=\frac{\exp[-s_b(x)/\tau]}{\sum_c\exp[-s_c(x)/\tau]}.$$
- $$\text{critical coordinate:}\quad \xi=\tau_{\mathrm{tie}}N^{\kappa/d},\qquad \pi_b\approx F_b(\xi).$$

## Implementation notes

1. Integration point: add this as a tie-resolution head on top of a continuous-action critic. Partition candidate actions into B local clusters using action-space distance or a learned mixture policy. For each cluster, estimate the immediate deficit and continuation transition distribution; use the resulting branch score during action selection and target construction. 2. Pseudocode: cluster candidate actions; for each branch b compute delta[b]=q_max-q_branch and obtain P_b either from a learned dynamics ensemble or by averaging next-state samples generated after executing representative actions; initialize z=0; repeat K times: for every state y compute m[y]=min_c(z[c,y]+delta[c,y]); update z_new[b,x]=delta[b,x]+beta*mean_y~P_b[m[y]]; stop when max_abs(z_new-z)<tol; form s[b]=z[b]+delta[b]; select with softmax(-s/tau), or use the branch's best candidate. 3. Computed versus estimated: contraction and iteration error are known analytically; delta, P_b, and the number of competing branches are estimated from the critic, replay data, and model ensemble. Use a soft-min during early training and anneal its temperature only after the fixed-point residual is small. 4. First cheap experiment: create a discounted gridworld or finite-state MDP with two action branches having identical immediate reward but different transition kernels, then add a controllable reward perturbation tau_tie and Gaussian critic noise. Compare ordinary argmax, random tie-breaking, and the tangential resolver in SAC-style sampled action selection. Sweep N and perturbation magnitude. The predicted signature is that the branch-selection curve collapses when plotted against tau_tie*N^(kappa/d), with a smooth transition near order one; ordinary argmax should switch at a noise-dependent location. Also verify geometric convergence of the inner solver with residual ratio beta, and reject the method if the observed ratio is not bounded by approximately beta plus estimation noise.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a finite-state Tangential Bellman Tie Resolver with value iteration, softmax branch selection, contraction checks, and reproducible scaling sweeps. The mechanism manifested: fixed-point error ratios matched beta across beta={0.2,0.5,0.8,0.95} to within about 5e-7, and resolver transition widths scaled exactly as 1/N, with width*N approximately 0.769 and normalized-curve deviation 6.4e-14. This is a strong toy-level confirmation, though not evidence of an RL performance win.

### Mechanism check

- Verdict: Built a finite-state Tangential Bellman Tie Resolver with value iteration, softmax branch selection, contraction checks, and reproducible scaling sweeps. The mechanism manifested: fixed-point error ratios matched beta across beta={0.2,0.5,0.8,0.95} to within about 5e-7, and resolver transition widths scaled exactly as 1/N, with width*N approximately 0.769 and normalized-curve deviation 6.4e-14. This is a strong toy-level confirmation, though not evidence of an RL performance win.
- Confidence: 8/10
- Limitations: Only a synthetic finite-state MDP was tested; no continuous-action critic, learned dynamics ensemble, SAC integration, replay data, environment benchmark, compute/FLOP comparison, or statistically repeated RL training was performed. The N-scaling experiment uses a controlled temperature rule tau=0.35/N and therefore verifies predicted mathematical scaling rather than estimating an empirical extreme-value exponent from sampled actions.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/785/bench_report.json)
- [bench_stage2.py](https://synthcore.org/code/785/bench_stage2.py)
- [report.md](https://synthcore.org/code/785/report.md)
- [report_bench_2026-09-01T103145.md](https://synthcore.org/code/785/report_bench_2026-09-01T103145.md)
- [results.json](https://synthcore.org/code/785/results.json)
- [tie_resolver_experiment.py](https://synthcore.org/code/785/tie_resolver_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/785)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.