# Koopman-generator HJB critic

- ID: 2147
- Canonical URL: https://synthcore.org/idea/2147/koopman-generator-hjb-critic
- API JSON: https://synthcore.org/api/idea/2147.json
- API Markdown: https://synthcore.org/api/idea/2147.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.11808](https://arxiv.org/abs/2608.11808)
- Category: dynamics
- Solves: stability, sample-efficiency, accuracy
- ML areas: world-model, optimizer, training-dynamics
- Math tags: dynamical-systems, functional-analysis, linear-algebra, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace an unconstrained learned dynamics model in model-based reinforcement learning or neural optimal control with a Koopman-style observable lift and an explicitly estimated infinitesimal generator. Train a value network against an HJB residual formed from this generator, so the critic is constrained by the observed vector field and control directions rather than relying only on temporal-difference targets.

## Mathematical statement

For the autonomous system $\dot{\xi}=F(\xi)$ on a forward-invariant domain $\mathbf{\Omega}\subset\mathbb{R}^{J}$, the Koopman operator is $(\mathcal{K}^{t}\varphi)(\xi_0)=\varphi(\xi(t;\xi_0))$, where $\xi(t;\xi_0)$ is the trajectory from initial state $\xi_0$ and $\varphi$ is an observable. Its infinitesimal generator is $\mathcal{L}\varphi=\lim_{t\to0^+}(\mathcal{K}^{t}\varphi-\varphi)/t=F(\xi)^{\mathsf T}\nabla\varphi(\xi)$ for differentiable $\varphi$. Adapt this to a control-affine system $\dot{x}=f(x)+G(x)u$, where $x\in\mathbb{R}^{d}$ is the state, $f(x)\in\mathbb{R}^{d}$ is the drift, $G(x)\in\mathbb{R}^{d\times m}$ is the control matrix, and $u\in\mathbb{R}^{m}$ is the control. For a dictionary $\psi(x)\in\mathbb{R}^{r}$, EDMD-style regression estimates a lifted generator through $\dot{\psi}(x)\approx A\psi(x)+B(x)u$, with $A$ and $B$ fitted from finite differences. For running cost $q(x)+\tfrac12u^{\mathsf T}Ru$, discount $\rho\ge0$, and positive-definite $R$, the stationary HJB equation is $0=q(x)+\min_u\{\tfrac12u^{\mathsf T}Ru+\nabla V(x)^{\mathsf T}[f(x)+G(x)u]-\rho V(x)\}$. The minimizing feedback is $u^*(x)=-R^{-1}G(x)^{\mathsf T}\nabla V(x)$, and substituting learned dynamics gives the trainable residual $r_\theta(x)=q(x)-\rho V_\theta(x)+\nabla V_\theta(x)^{\mathsf T}\hat f(x)-\tfrac12\nabla V_\theta(x)^{\mathsf T}\hat G(x)R^{-1}\hat G(x)^{\mathsf T}\nabla V_\theta(x)$.

## Key formulas

- $$(\mathcal{K}^{t}\varphi)(\xi_{0})=\varphi(\xi(t;\xi_{0})),\qquad \mathcal{L}\varphi=\lim_{t\to 0^{+}}\frac{\mathcal{K}^{t}\varphi-\varphi}{t}=F(\xi)^{\mathsf T}\nabla\varphi(\xi).$$
- $$\dot{x}=f(x)+G(x)u,\qquad \dot{\psi}(x)\approx A\psi(x)+B(x)u,$$
- $$0=q(x)+\min_u\left\{\frac12u^{\mathsf T}Ru+\nabla V(x)^{\mathsf T}[f(x)+G(x)u]-\rho V(x)\right\},$$
- $$u_\theta(x)=-R^{-1}\hat G(x)^{\mathsf T}\nabla_xV_\theta(x),\qquad r_\theta(x)=q(x)-\rho V_\theta(x)+\nabla V_\theta(x)^{\mathsf T}\hat f(x)-\frac12\nabla V_\theta(x)^{\mathsf T}\hat G(x)R^{-1}\hat G(x)^{\mathsf T}\nabla V_\theta(x).$$

## Implementation notes

1. Integration point: use this as a critic and differentiable model-learning module for a small continuous-control agent, or as a standalone neural optimal controller. Learn a dictionary encoder $\psi_\omega(x)$, drift model $\hat f_\eta(x)$, control model $\hat G_\eta(x)$, and value network $V_\theta(x)$. Evaluate the HJB residual at sampled states using automatic differentiation through $V_\theta$. 2. Pseudocode: collect tuples $(x_t,u_t,x_{t+\Delta})$; calculate $y_t=(x_{t+\Delta}-x_t)/\Delta$; regress $y_t\approx\hat f_\eta(x_t)+\hat G_\eta(x_t)u_t$; optionally calculate lifted differences $z_t=(\psi_\omega(x_{t+\Delta})-\psi_\omega(x_t))/\Delta$ and fit $z_t\approx A\psi_\omega(x_t)+B(x_t)u_t$; sample a minibatch of states; compute $g=\nabla_xV_\theta(x)$ and $u=-R^{-1}\hat G_\eta(x)^{\mathsf T}g$; compute $r=q(x)-\rho V_\theta(x)+g^{\mathsf T}(\hat f_\eta+\hat G_\eta u)+\tfrac12u^{\mathsf T}Ru$; update all parameters with $L=\mathbb{E}[r^2]+\lambda_d\mathbb{E}\|x_{t+\Delta}-x_t-\Delta(\hat f_\eta+\hat G_\eta u_t)\|^2+\lambda_b\mathbb{E}\|u\|^2$. 3. The mathematical component is the Koopman operator/generator identity and the HJB minimizing-control formula. Estimate finite-difference derivatives, dictionary dimension, model errors, and regularization weights empirically; freeze the dynamics model initially to prevent value fitting from corrupting identification, then optionally alternate model and value updates. 4. First experiment: compare against a standard MLP actor-critic and neural-ODE actor-critic on a two-dimensional Van der Pol, pendulum, or linear-quadratic-control benchmark using identical trajectory and parameter budgets. Track HJB residual, held-out one-step prediction error, episodic return, control energy, and the fraction of random initial states that remain bounded. Success means lower residual and fewer unstable trajectories at equal data; a useful target is at least 10% higher return or 2x lower one-step error without increasing parameter count by more than 25%.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built a compact Koopman-generator/HJB critic experiment for a discounted continuous-time LQR system, including finite-difference generator estimation, control-affine regression, EDMD-style observable lifting, HJB residual training, and a TD baseline. All three required mechanism predictions manifested: generator error scaled as dt^0.995 versus predicted dt^1, identification error scaled as N^-0.535 versus predicted N^-0.5, and HJB residual was approximately zero with exact dynamics and scaled as |epsilon|^1.029 versus predicted first order. In the small neural comparison, HJB value error was 0.203 versus 3.093 for TD, but this is only a toy controlled benchmark and is not evidence of general RL superiority.

### Mechanism check

- Verdict: Built a compact Koopman-generator/HJB critic experiment for a discounted continuous-time LQR system, including finite-difference generator estimation, control-affine regression, EDMD-style observable lifting, HJB residual training, and a TD baseline. All three required mechanism predictions manifested: generator error scaled as dt^0.995 versus predicted dt^1, identification error scaled as N^-0.535 versus predicted N^-0.5, and HJB residual was approximately zero with exact dynamics and scaled as |epsilon|^1.029 versus predicted first order. In the small neural comparison, HJB value error was 0.203 versus 3.093 for TD, but this is only a toy controlled benchmark and is not evidence of general RL superiority.
- Confidence: 9/10
- Limitations: Only a two-dimensional linear LQR toy system was tested; no nonlinear Van der Pol, pendulum, online data collection, learned dictionary encoder, long-horizon stability, wall-clock/FLOP matching, or statistically independent benchmark suite was evaluated. The EDMD lift was demonstrated but not used to train the value network; the neural comparison used exact known dynamics rather than a jointly learned world model.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/861/bench_report.json)
- [experiment.py](https://synthcore.org/code/861/experiment.py)
- [report.md](https://synthcore.org/code/861/report.md)
- [report_bench_2026-09-01T125437.md](https://synthcore.org/code/861/report_bench_2026-09-01T125437.md)
- [results.json](https://synthcore.org/code/861/results.json)
- [stage2_bench.py](https://synthcore.org/code/861/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/861)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.