# Slow-MPC Fast-Policy Residual Control

- ID: 2636
- Canonical URL: https://synthcore.org/idea/2636/slow-mpc-fast-policy-residual-control
- API JSON: https://synthcore.org/api/idea/2636.json
- API Markdown: https://synthcore.org/api/idea/2636.md
- Verification status: unverified
- Source: [arXiv:2608.20858](https://arxiv.org/abs/2608.20858)
- Category: dynamics
- Solves: stability, speedup, scalability
- ML areas: rl, optimizer, training-dynamics, world-model
- Math tags: control-theory, dynamical-systems, optimization, numerical-analysis
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Split a neural controller into a slow model-based planner and a fast policy instead of requiring either component to perform the entire control task. The MPC output provides a slowly varying nominal action or operating envelope, while the neural policy generates high-frequency residual corrections. This should preserve constraint handling while reducing the frequency of expensive online optimization.

## Mathematical statement

Let $z_k$ be the environment state at fast time index $k$, let $T_l$ be the neural control interval, let $T_h=M T_l$ be the slower MPC interval, and let $M\in\mathbb{N}$ be the time-scale ratio. Every $M$ fast steps, solve a finite-horizon MPC problem using an approximate model $\hat f$, obtaining a nominal action $u^{\mathrm{mpc}}_j$. At every fast step, the neural policy $\pi_\theta$ receives the normalized state, exogenous input, and held MPC command, and produces a residual $\delta u_k$. The applied action is $u_k=\Pi_{\mathcal U(z_k)}(u^{\mathrm{mpc}}_{\lfloor k/M\rfloor}+\alpha\delta u_k)$, where $\alpha$ is a residual scale and $\Pi_{\mathcal U(z_k)}$ projects onto admissible actions. The transferable mechanism is time-scale separation: expensive constrained optimization is amortized over $M$ fast steps while the policy handles local high-frequency corrections. The paper supplies no general Lyapunov guarantee; the falsifiable prediction is that increasing $M$ reduces MPC computation approximately in proportion to $1/M$ until stale nominal commands cause a sharp increase in constraint violations or task cost.

## Key formulas

- $$u^{\mathrm{mpc}}_j=\arg\min_{u_{j:j+H-1}}\sum_{r=0}^{H-1}\ell(\hat z_{j+r|j},u_{j+r})+\ell_f(\hat z_{j+H|j})$$
$$\text{subject to }\hat z_{j+r+1|j}=\hat f(\hat z_{j+r|j},u_{j+r}),\quad \hat z_{j+r|j}\in\mathcal X,\quad u_{j+r}\in\mathcal U,$$
where $H$ is the MPC horizon, $\ell$ is the stage cost, $\ell_f$ is the terminal cost, $\mathcal X$ is the state-constraint set, and $\mathcal U$ is the action-constraint set.
- $$\delta u_k=\pi_\theta\!\left(\operatorname{norm}(z_k),\operatorname{norm}(d_k),u^{\mathrm{mpc}}_{\lfloor k/M\rfloor}\right),\qquad u_k=\Pi_{\mathcal U(z_k)}\left(u^{\mathrm{mpc}}_{\lfloor k/M\rfloor}+\alpha\delta u_k\right),$$
where $d_k$ denotes exogenous inputs, $\operatorname{norm}(\cdot)$ is feature normalization, $\alpha$ is the residual-gain parameter, and $\Pi_{\mathcal U(z_k)}$ projects onto the feasible action set.
- $$\mathrm{cost}(M)=\mathrm{cost}_{\mathrm{policy}}+\frac{1}{M}\mathrm{cost}_{\mathrm{MPC}}+\mathrm{staleness\_penalty}(M),$$
where the first term is fast policy computation, the second is the amortized MPC cost, and the final term captures degradation caused by holding the nominal action for $M$ fast steps. Define $M_{\mathrm{crit}}$ as the smallest ratio for which constraint violations or task cost increase sharply.

## Implementation notes

1. Exact integration point: use this controller for continuous-control RL, model-based RL, or a learned-world-model environment. Partition the action into a slow nominal component and a fast residual component, or use the MPC output as an additional policy input. Normalize state and exogenous-input features. Run MPC only when $k\bmod M=0$ and hold its nominal command between solves; evaluate the policy at every environment step. 2. Pseudocode: initialize $u_{\mathrm{mpc}}=0$; for every fast step $k$, observe $z_k,d_k$; if $k\bmod M=0$, roll out $\hat f$, solve the constrained horizon problem, and set $u_{\mathrm{mpc}}$ to its first action; compute $\delta u=\pi_\theta([\operatorname{norm}(z_k),\operatorname{norm}(d_k),u_{\mathrm{mpc}}])$; apply $u_k=\Pi_{\mathcal U(z_k)}(u_{\mathrm{mpc}}+\alpha\delta u)$; store the transition and update the policy with PPO, SAC, or an actor-critic objective. Detach the MPC solution during policy training initially. 3. Computed versus estimated: MPC trajectories, nominal actions, and projections are computed directly; model mismatch, the staleness penalty, and $M_{\mathrm{crit}}$ are estimated from rollouts. 4. First cheap experiment: compare SAC, MPC, and the hybrid controller on Pendulum or a small constrained linear system with injected model mismatch. Sweep $M\in\{1,2,4,8,16\}$ and $\alpha\in\{0.1,0.3,1\}$. Measure return, wall-clock MPC calls, constraint violations, and action power spectra. The quantitative prediction is that MPC calls and MPC wall-clock cost fall approximately as $1/M$, low-frequency action power remains concentrated below the MPC update frequency, and high-frequency power is supplied by the policy. Performance should show a knee at $M_{\mathrm{crit}}$: beyond it, stale nominal actions cause a sharp rise in violations or cost. An initial acceptance criterion is at least 50% fewer MPC calls with no more than 5% return loss and no increase in violation rate before the measured knee.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.