# Contraction-budgeted MPPI policy head

- ID: 505
- Canonical URL: https://synthcore.org/idea/505/contraction-budgeted-mppi-policy-head
- API JSON: https://synthcore.org/api/idea/505.json
- API Markdown: https://synthcore.org/api/idea/505.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.06945](https://arxiv.org/abs/2607.06945)
- Category: dynamics
- Solves: stability, sample-efficiency, accuracy
- ML areas: world-model, rl, rnn, ssm
- Math tags: control-theory, dynamical-systems, probability, optimization
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Attach a sampling-based rollout correction head to a neural policy or learned world model, and adapt its temperature and number of rollouts so that approximation error stays within the contraction margin of a nominal policy. The controller should spend samples only when the local state-dependent error gain is close to violating the small-gain condition, instead of using a fixed MPPI sample count everywhere.

## Mathematical statement

The paper decomposes MPPI action approximation error as \(\|u^{\mathrm{MPPI}}_k-\pi^*(x_k)\|\leq \beta_\infty\|x_k-x^*\|+e_M(\eta)\), with \(e_M(\eta)=\beta_0+\varepsilon_M(\eta)\). Here \(u^{\mathrm{MPPI}}_k\) is the sampled controller action, \(\pi^*(x_k)\) is a nominal stabilizing policy, \(x_k\) is the current state, \(x^*\) is the equilibrium, \(\beta_\infty\) is the state-dependent approximation gain, \(\beta_0\) is the infinite-sample finite-temperature bias, and \(\varepsilon_M(\eta)\) is the finite-sample Monte Carlo error for \(M\) rollouts and confidence parameter \(\eta\). The paper gives \(\varepsilon_M(\eta)=O(M^{-1/2})\), while lowering MPPI temperature \(\lambda\) reduces \(\beta_0\). Its small-gain condition is \(\Phi(\beta_\infty)=\sqrt{\bar\mu/\mu}\,L_u\beta_\infty\leq(1-\beta)/2\), where \(\mu\) and \(\bar\mu\) are lower and upper bounds on the contraction metric, \(L_u\) is the dynamics' local Lipschitz constant with respect to action, and \(\beta<1\) is the nominal contraction factor. The neural adaptation uses a learned policy \(\pi_\theta\) as the nominal reference and enforces \(\sqrt{\bar\mu/\mu}L_u\widehat\beta_\infty+\widehat e_M\leq(1-\widehat\beta)/2\).

## Key formulas

- $$\left\|u^{\mathrm{MPPI}}_{k}-\pi^{*}(x_{k})\right\|\leq\beta_{\infty}\left\|x_{k}-x^{*}\right\|+e_{M}(\eta),\qquad e_M(\eta):=\beta_0+\varepsilon_M(\eta),\qquad \varepsilon_M(\eta)=O(M^{-1/2}).$$
- $$\Phi(\beta_{\infty})=\sqrt{\frac{\bar{\mu}}{\mu}}\,L_{u}\,\beta_{\infty}\leq\frac{1-\beta}{2}.$$
- $$\sqrt{\frac{\bar\mu}{\mu}}L_u\widehat\beta_\infty+\widehat e_M(\eta)\leq\frac{1-\widehat\beta}{2},\qquad \widehat e_M(\eta)=\widehat\beta_0(\lambda)+\widehat c\sqrt{\frac{\log(1/\eta)}{M}}.$$
- $$M\geq\frac{\widehat c^2\log(1/\eta)}{\left((1-\widehat\beta)/2-\sqrt{\bar\mu/\mu}\,L_u\widehat\beta_\infty-\widehat\beta_0(\lambda)\right)^2}.$$

## Implementation notes

(1) Integration point: use a neural policy \(\pi_\theta(x)\) inside a model-based RL or world-model controller. At every state, generate action perturbations around the network's proposed action, roll them through a learned dynamics model, score each trajectory, and form an MPPI-weighted action. The nominal reference is \(\pi_\theta\); the output is the corrected action \(u^{\mathrm{MPPI}}\). This is primarily an inference-time module, with optional training losses for improving the contraction margin.

(2) Pseudocode:
```
input x, policy pi_theta, dynamics f_phi, temperature lambda
a0 = pi_theta(x)
for j = 1,...,M:
    sample delta_j[0:H-1]
    set a_j[t] = pi_theta(x_j[t]) + delta_j[t]
    roll out x_j[t+1] = f_phi(x_j[t], a_j[t])
    compute trajectory cost J_j
weights w_j = softmax(-J_j / lambda)
u = pi_theta(x) + sum_j w_j * delta_j[0]
estimate beta_inf, L_u, and beta from recent transitions
e_M = beta0_hat(lambda) + c_hat * sqrt(log(1/eta) / M)
if sqrt(mu_bar/mu)*L_u*beta_inf_hat + e_M > (1-beta_hat)/2:
    increase M, or decrease lambda if beta0_hat decreases
execute u
```

(3) Estimate the mathematical quantities empirically. Estimate \(L_u\) with finite differences through the learned dynamics: \(\widehat L_u=\max_i\|f_\phi(x_i,u_i+\delta u_i)-f_\phi(x_i,u_i)\|/\|\delta u_i\|\). Estimate \(\widehat\beta_\infty\) as a robust upper-quantile slope of \(\|u^{\mathrm{MPPI}}-\pi_\theta(x)\|\) versus \(\|x-x^*\|\). Estimate \(\widehat\beta\) from the 95th-percentile ratio of successive state distances under the nominal policy. Estimate \(\widehat c\) by running independent rollout batches and fitting action-estimate standard deviation against \(1/\sqrt M\). Estimate \(\widehat\beta_0(\lambda)\) using a very large rollout batch at each candidate temperature. Initially use Euclidean state distance, so \(\bar\mu/\mu=1\); later replace it with bounds from a learned positive-definite metric.

(4) First cheap experiment: use a 2-layer MLP policy and a small learned ensemble dynamics model on Pendulum-v1 or CartPole swing-up. Compare fixed-sample MPPI, fixed-temperature MPPI, and this adaptive controller at equal average rollout FLOPs. Measure success rate, action variance, catastrophic departures from a safe state set, reward, and executed rollouts. The expected signal is fewer unstable episodes and lower action variance at the same average sample count, with adaptive rollout increases concentrated near difficult states. At equal failure rate, it should require fewer rollouts than a conservative fixed-sample baseline.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built a contraction-budgeted MPPI toy controller with adaptive rollout allocation and numerical checks of the claimed mechanism. The predicted Monte Carlo exponent (-0.5) was observed as -0.523, the quadratic temperature-bias formula matched closely with bias/temperature correlation 0.996, the small-gain boundary was exactly detected at beta_inf=0.15, and budget size scaled with margin^-2 (observed exponent -1.988). The adaptive controller used 49.1 versus 128 mean rollouts with zero failures, but had worse toy reward (-1.188 versus -0.814), so the mechanism manifested without an overall performance win in this setup.

### Mechanism check

- Verdict: Built a contraction-budgeted MPPI toy controller with adaptive rollout allocation and numerical checks of the claimed mechanism. The predicted Monte Carlo exponent (-0.5) was observed as -0.523, the quadratic temperature-bias formula matched closely with bias/temperature correlation 0.996, the small-gain boundary was exactly detected at beta_inf=0.15, and budget size scaled with margin^-2 (observed exponent -1.988). The adaptive controller used 49.1 versus 128 mean rollouts with zero failures, but had worse toy reward (-1.188 versus -0.814), so the mechanism manifested without an overall performance win in this setup.
- Confidence: 9/10
- Limitations: This is an analytically structured scalar toy rather than Pendulum, CartPole, a neural policy, or a learned world model. It does not test real wall-clock/FLOP performance, learned estimates of beta_inf/L_u/beta, confidence calibration, process-noise robustness beyond the toy noise model, or whether adaptive sampling improves reward at matched rollout budget.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.1861
- Paired wins: 2/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/241/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/241/bench_report.json)
- [experiment.py](https://synthcore.org/code/241/experiment.py)
- [report.md](https://synthcore.org/code/241/report.md)
- [report_bench_2026-08-30T212730.md](https://synthcore.org/code/241/report_bench_2026-08-30T212730.md)
- [results.json](https://synthcore.org/code/241/results.json)
- [Download all files as ZIP](https://synthcore.org/download/241)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.