# Innovation-Compensated Latent Policy

- ID: 548
- Canonical URL: https://synthcore.org/idea/548/innovation-compensated-latent-policy
- API JSON: https://synthcore.org/api/idea/548.json
- API Markdown: https://synthcore.org/api/idea/548.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.07276](https://arxiv.org/abs/2607.07276)
- Category: architecture
- Solves: stability, accuracy, sample-efficiency
- ML areas: world-model, rl, rnn, training-dynamics
- Math tags: dynamical-systems, control-theory, differential-geometry
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

In a partially observed reinforcement-learning or model-based control agent, expose the state-estimator innovation to the action head through a dedicated residual feedback branch. The policy produces a nominal action from the estimated latent state, while a learned innovation-compensation branch corrects actions when observations disagree with predicted latent dynamics. This explicitly separates nominal policy behavior from estimation-induced corrections and should help during fast transients, sensor dropouts, and distribution shifts.

## Mathematical statement

The plant model is $\dot{\bm{x}}=\bm{f}(\bm{x})+\mathbf{g}(\bm{x})\bm{u}+\bm{w}$ and $\bm{y}=\bm{h}(\bm{x})+\bm{v}$, where $\bm{x}$ is the hidden state, $\bm{u}$ the control, $\bm{y}$ the observation, and $\bm{w},\bm{v}$ process and measurement disturbances. Equation (33) decomposes estimated tracking-error dynamics as $\dot{\hat{\bm{\epsilon}}}_{\bm{x}}=(\bm{F}_{\hat{\bm{x}}}-\bm{G}_{\hat{\bm{x}}}\mathbf{K})\hat{\bm{\epsilon}}_{\bm{x}}+\bm{\Omega}(\tilde{\bm{x}},\bm{w},\bm{v})+\bm{\delta}(\bm{x},\hat{\bm{x}},\bm{u})$, where $\hat{\bm{\epsilon}}_{\bm{x}}$ is estimated tracking error, $\bm{F}_{\hat{\bm{x}}}$ and $\bm{G}_{\hat{\bm{x}}}$ are local Jacobians of the drift and input maps, $\mathbf{K}$ is feedback gain, $\bm{\Omega}$ is innovation coupling caused by estimation residual $\tilde{\bm{x}}=\bm{x}-\hat{\bm{x}}$ and disturbances, and $\bm{\delta}$ is the nonlinear structural residual. Equation (31) compensates this coupling using $\bm{u}=\mathbf{g}^{\dagger}(\hat{\bm{x}})(\dot{\bm{x}}_{\mathrm{ref}}-\mathbf{K}\hat{\bm{\epsilon}}_{\bm{x}}-\bm{f}(\hat{\bm{x}})-\bm{L}_{\hat{\bm{x}}}\tilde{\bm{y}})$, where $\mathbf{g}^{\dagger}$ is a control-effectiveness pseudoinverse, $\bm{L}_{\hat{\bm{x}}}$ maps innovation to control, and $\tilde{\bm{y}}$ is the observation innovation. The neural adaptation preserves the additive structure as $u_t=\pi_\theta(\hat{z}_t)-B_\phi(\hat{z}_t)K e_t-L_\phi(\hat{z}_t)\nu_t$. The tube expression $\Delta\mathcal{T}_{\epsilon}=\|\tilde{\bm{x}}\|(\sqrt{\rho^{-1}(1+d/(\mu\|\tilde{\bm{x}}\|^2))}-1)$, with $\rho=\lambda/\mu$, shows that increasing nominal dissipation without reducing perception error can produce a negative tracking margin; therefore the innovation branch is gated and gain-limited by uncertainty.

## Key formulas

- $$\dot{\bm{x}}=\bm{f}(\bm{x})+\mathbf{g}(\bm{x})\bm{u}+\bm{w},\qquad \bm{y}=\bm{h}(\bm{x})+\bm{v}.$$
- $$\dot{\hat{\bm{\epsilon}}}_{\bm{x}}=(\bm{F}_{\hat{\bm{x}}}-\bm{G}_{\hat{\bm{x}}}\mathbf{K})\hat{\bm{\epsilon}}_{\bm{x}}+\bm{\Omega}(\tilde{\bm{x}},\bm{w},\bm{v})+\bm{\delta}(\bm{x},\hat{\bm{x}},\bm{u}).$$
- $$\bm{u}=\mathbf{g}^{\dagger}(\hat{\bm{x}})\bigl(\dot{\bm{x}}_{\mathrm{ref}}-\mathbf{K}\hat{\bm{\epsilon}}_{\bm{x}}-\bm{f}(\hat{\bm{x}})-\bm{L}_{\hat{\bm{x}}}\tilde{\bm{y}}\bigr).$$
- $$u_t=\pi_\theta(\hat{z}_t)-\alpha_t B_\phi(\hat{z}_t)K e_t-\alpha_t L_\phi(\hat{z}_t)\nu_t,\qquad \alpha_t=\operatorname{clip}\!\left(\frac{1}{1+c\,\widehat{\sigma}_t},0,1\right).$$

## Implementation notes

(1) Exact integration point: use a recurrent latent-state estimator inside a partially observed RL or model-based control policy. At time $t$, feed observation $o_t$ and previous action $u_{t-1}$ to an estimator, which returns latent state $\hat z_t$, predicted observation $\hat o_t$, recurrent state $h_t$, and uncertainty estimate $\widehat\Sigma_t$. Define innovation $\nu_t=o_t-\hat o_t$. The action head has three explicit paths: nominal policy $\pi_\theta(\hat z_t)$, tracking-error feedback $B_\phi(\hat z_t)K e_t$, and innovation compensation $L_\phi(\hat z_t)\nu_t$. For tasks without a reference trajectory, set $e_t=0$ or use $e_t=\hat z_t-z_t^{\mathrm{goal}}$.

(2) Pseudocode:
```
(h_t, zhat_t, ohat_t, Sigma_t) = estimator(o_t, u_prev, h_prev)
nu_t = o_t - ohat_t
unc_t = mean(diag(Sigma_t))
alpha_t = clip(1 / (1 + c * unc_t), 0, 1)
u_nom = policy(zhat_t)
u_fb = B_phi(zhat_t) @ K @ e_t
nu_ic = L_phi(zhat_t) @ nu_t
u_t = tanh(u_nom - alpha_t * (nu_fb + nu_ic))
```
Train with the task return or supervised control loss, estimator prediction loss $\|o_t-\hat o_t\|^2$, and a small penalty on $\|L_\phi\nu_t\|^2$.

(3) Compute the additive decomposition and innovation input from equations (31) and (33). Learn $B_\phi$ and $L_\phi$ by backpropagation; use fixed or learned diagonal $K$ with spectral-norm clipping. Estimate $\widehat\Sigma_t$ using an ensemble of estimator heads, a diagonal variational covariance, or an exponential moving average of innovation magnitudes. Set $c$ on validation data and clip the operator norm of $L_\phi$ to avoid high-gain instability.

(4) First experiment: implement the module in a small GRU policy on partially observed MuJoCo Hopper, HalfCheetah, or a quadrotor trajectory-tracking simulator with randomized sensor noise, observation dropouts, and fast reference changes. Compare a standard GRU policy, a latent-state policy without innovation input, and the proposed policy using identical parameter counts and PPO or SAC budgets. Measure return, RMS tracking error, recovery time after observation corruption, action variance, and failure rate. The key falsifiable signal is lower transient tracking error and faster recovery at equal environment steps, with no clean-observation degradation. Ablate the innovation branch, uncertainty gate, estimator prediction loss, and gain clipping separately.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a recurrent latent-state control MVP with an explicit nominal path, learned innovation-compensation path, prediction loss, and uncertainty gate. The gate behaved as claimed numerically: alpha decreased monotonically from 1.0 at zero uncertainty to 0.0476 at uncertainty 10 while remaining in [0,1]. However, in the matched partially observed linear-control experiment, innovation compensation slightly improved clean transient error (0.187 vs 0.206) but worsened clean RMS error (0.284 vs 0.277) and corruption recovery error (0.299 vs 0.274), so the promised robust win was not observed.

### Mechanism check

- Verdict: Built a recurrent latent-state control MVP with an explicit nominal path, learned innovation-compensation path, prediction loss, and uncertainty gate. The gate behaved as claimed numerically: alpha decreased monotonically from 1.0 at zero uncertainty to 0.0476 at uncertainty 10 while remaining in [0,1]. However, in the matched partially observed linear-control experiment, innovation compensation slightly improved clean transient error (0.187 vs 0.206) but worsened clean RMS error (0.284 vs 0.277) and corruption recovery error (0.299 vs 0.274), so the promised robust win was not observed.
- Confidence: 9/10
- Limitations: Only one small fixed-seed benchmark and short training budget were tested; the plant is linear rather than MuJoCo/nonlinear, no PPO or multi-seed statistical analysis was performed, and the estimator used a learned observation predictor rather than a full learned dynamics model or ensemble covariance.

## Artifacts

- [innovation_experiment.py](https://synthcore.org/code/1382/innovation_experiment.py)
- [report.md](https://synthcore.org/code/1382/report.md)
- [results.json](https://synthcore.org/code/1382/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1382)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.