# Worst-case universal excitation for world-model training

- ID: 315
- Canonical URL: https://synthcore.org/idea/315/worst-case-universal-excitation-for-world-model-training
- API JSON: https://synthcore.org/api/idea/315.json
- API Markdown: https://synthcore.org/api/idea/315.md
- Verification status: unverified
- Source: [arXiv:2607.00844](https://arxiv.org/abs/2607.00844)
- Category: training
- Solves: sample-efficiency, accuracy, stability
- ML areas: world-model, rnn, data-augmentation, training
- Math tags: control-theory, dynamical-systems, linear-algebra, optimization
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Replace random exploration or generic input noise with interventions selected to separate the predictions of all plausible dynamics models in an ensemble. At each data-collection step, choose the bounded input sequence whose simulated trajectories produce the largest worst-case information matrix, while penalizing unsafe or high-energy actions. This should make a recurrent or state-space world model identify dynamics with fewer real trajectories and reduce uncertainty on long-horizon rollouts.

## Mathematical statement

The paper studies the discrete-time linear model $x(t+1)=Ax(t)+Bu(t)+w(t)$ with bounded noise $\|w(t)\|\leq\varepsilon$, where $x(t)\in\mathbb{R}^{n}$ is the state, $u(t)\in\mathbb{R}^{m}$ is the input, $(A,B)$ are unknown dynamics matrices, and $w(t)$ is disturbance. For a sequence $z_{[a,b]}$, define the depth-one Hankel matrix $\mathcal{H}_{1}(z_{[a,b]})=[z(a)\;z(a+1)\;\cdots\;z(b)]$. The extracted set-membership certificate is $\|\mathcal{H}_{1}(x_{[1,T]})-A\mathcal{H}_{1}(x_{[0,T-1]})-B\mathcal{H}_{1}(u_{[0,T-1]})\|\leq\varepsilon$, which means that $(A,B)$ is compatible with the data under the bounded-noise model. The associated quadratic form is $\begin{bmatrix}I&A^{\top}&B^{\top}\end{bmatrix}RR^{\top}\begin{bmatrix}I\\A^{\top}\\B^{\top}\end{bmatrix}\leq\varepsilon^{2}I_n$, where $R=\begin{bmatrix}\mathcal{H}_{1}(x_{[1,T]})\\-\mathcal{H}_{1}(x_{[0,T-1]})\\-\mathcal{H}_{1}(u_{[0,T-1]})\end{bmatrix}$. Adapt this by representing the learned dynamics ensemble as $\theta_j=(A_j,B_j)$ and maximizing the minimum pairwise informativeness of the input-state regressor $\Phi_j=[\mathcal{H}_1(x_{[0,T-1]});\mathcal{H}_1(u_{[0,T-1]})]$. A practical robust objective is $J(u)=\min_j\sigma_{\min}(\Phi_j^{\top}\Phi_j+\lambda I)-\beta\sum_t\|u_t\|^2$, where $\Phi_j$ is generated by model $j$, $\lambda>0$ is numerical regularization, and $\beta$ controls input energy.

## Key formulas

- $$x(t+1)=Ax(t)+Bu(t)+w(t),\qquad \|w(t)\|\leq\varepsilon.$$
- $$\left\|\mathcal{H}_{1}(x_{[1,T]})-A\mathcal{H}_{1}(x_{[0,T-1]})-B\mathcal{H}_{1}(u_{[0,T-1]})\right\|\leq\varepsilon.$$
- $$\Phi=\begin{bmatrix}\mathcal{H}_{1}(x_{[0,T-1]})\\\mathcal{H}_{1}(u_{[0,T-1]})\end{bmatrix},\qquad J(u_{0:T-1})=\min_{j\in[1,K]}\sigma_{\min}\!\left(\Phi_j^{\top}\Phi_j+\lambda I\right)-\beta\sum_{t=0}^{T-1}\|u_t\|^2.$$
- $$u^*=\arg\max_{u_{0:T-1}\in\mathcal{U}}J(u),\qquad \mathcal{U}=\{u_{0:T-1}:\|u_t\|\leq u_{\max}\}.$$

## Implementation notes

Integrate this at the data-collection or rollout-planning interface of a recurrent world model, neural state-space model, or model-based RL agent. Maintain an ensemble of K models with identical architecture but different bootstrap samples, checkpoints, or stochastic parameters. For a candidate action sequence $u_{0:T-1}$, start from the current observed state $x_0$, roll out every ensemble member, and construct $\Phi_j=[x_0,\ldots,x_{T-1};u_0,\ldots,u_{T-1}]$ as a stacked matrix. Estimate $\sigma_{\min}(\Phi_j^\top\Phi_j+\lambda I)$ with an eigensolver or a few inverse or power iterations. Optimize the candidate sequence using CEM, random shooting, or differentiable gradient ascent on the smooth surrogate $\operatorname{softmin}_j s_j=-\tau\log\sum_j\exp(-s_j/\tau)$, where $s_j=\sigma_{\min}(\Phi_j^\top\Phi_j+\lambda I)$. Subtract the energy penalty and reject sequences violating action or state constraints. Execute only the first action, append the real transition to the replay buffer, retrain the ensemble, and replan. The paper-derived computation is the bounded-noise compatibility residual and the singular-value informativeness criterion; ensemble rollout uncertainty and the values of $\beta$ and $\tau$ are empirical design choices. Begin with a 2D linear system and compare random Gaussian actions, sinusoidal persistent excitation, and the proposed worst-case planner using the same action budget. Then test a GRU world model on CartPole or Pendulum. Measure held-out one-step error, multi-step rollout error, the smallest singular value of the accumulated regressor, and the number of transitions required to reach a fixed prediction-error threshold. Success is a larger regressor singular value and at least 20 percent fewer environment transitions at equal action bounds, without increased rollout instability.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.