Global-Local Koopman Latent Dynamics

(1) Integration point: use this as the latent transition inside a small video-prediction, robot-world-model, or sequence-model backbone. Given an observation window $x_{t-r+1:t}$, implement two encoders: a global encoder $\phi_g$ producing $z^g_t\in\mathbb{R}^{d_g}$ and a shared local encoder $\phi_l$ producing $z^l_t\in\mathbb{R}^{N\times d_l}$ for patches, objects, graph nodes, or body segments. Flatten local latents for the first MVP. Replace the usual GRU or MLP transition with the block update from the formula, and use a decoder $D([z^g;z^l])$ to reconstruct the next observation. (2) Pseudocode: encode $(z_g,z_l)=(\phi_g(x_{t-r+1:t}),\phi_l(x_{t-r+1:t}))$; for $j=0,\ldots,H-1$, compute $z_g\leftarrow A_gz_g+B_gu_{t+j}$ and $z_l\leftarrow A_lz_l+B_lu_{t+j}$, then $\hat{x}_{t+j+1}\leftarrow D([z_g;z_l])$; minimize the displayed horizon loss, with $P_g$ selecting downsampled or global features and $P_l$ selecting local patches or segments. (3) Estimate $A_g,A_l,B_g,B_l$ either as trainable matrices or by least squares on detached encoder outputs. Initialize each $A$ near identity and optionally constrain its largest singular value to at most $1+\epsilon$ using spectral normalization or projection. The zero cross-block structure comes from the paper; nonlinear coupling is learned by the encoders and decoder. $H$ is the rollout horizon, $r$ is the history length, $N$ is the number of local entities, and $\lambda_g,\lambda_l,\beta$ are validation-tuned weights. (4) First cheap experiment: train on Moving-MNIST, bouncing balls, or a small robot-arm trajectory dataset using a two-layer GRU latent baseline with the same parameter count and rollout horizon. Compare one-step and 20-step prediction error, gradient norm growth, and rollout wall-clock time. Success is lower long-horizon MSE and less error explosion at equal FLOPs; for a control dataset, additionally test whether a simple shooting planner using the linear latent rollout achieves lower trajectory-tracking error without retraining under altered payload or disturbance conditions.