Dual Information-Demand Curiosity

(1) Integration point: modify the loss of a probabilistic latent world model used by model-based RL. Let an encoder produce z_t, a recurrent transition model produce z_{t+1:t+K} and predicted parameter embeddings theta=(A,B), and a decoder produce future-observation likelihood q_phi(y_{t+1:t+K}|z_{t+1:t+K},theta). Keep the policy/action distribution q(u_{t+1:t+K}) unchanged initially. Replace a fixed curiosity coefficient with the dual term lambda*(H_goal-I_hat).

(2) Pseudocode: sample N trajectory starts and imagined action sequences u. Unroll the transition model to obtain z and theta, then sample or reparameterize predicted observations y. Compute the usual Bethe/ELBO world-model loss F_B, including local transition/emission factor terms and entropy corrections. Train q_phi(y|z,theta,u) and a separate q_psi(y|u) marginal decoder. Compute I_hat = mean(log q_phi(y|z,theta,u)-log q_psi(y|u)); detach I_hat for the dual update but not for the primal loss. Minimize F_B + lambda*(H_goal-I_hat) with respect to model parameters. Update lambda after each batch as lambda=max(0,lambda+eta_lambda*(H_goal-EMA(I_hat))); optionally clip lambda to lambda_max.

(3) Computed from the mathematics: the sign of the Lagrangian term, the nonnegative dual update, complementary-slackness behavior, and the Bethe entropy-counting correction. Estimated empirically: I_hat, the EMA used for its constraint, and H_goal when the goal prior is learned. For stability, use stop-gradient on q_psi targets, a slowly updated EMA of I_hat, gradient clipping, and warm-start lambda=0 for the first few thousand updates. Test three regimes by setting H_goal below, near, and above the model's attainable mutual information; the expected behavior is respectively lambda approximately zero, an interior adaptive lambda, and lambda approaching its clip.

(4) First experiment: use a small RSSM or transformer world model on MiniGrid or DMControl with sparse visual observations, comparing standard ELBO plus fixed curiosity, unconstrained mutual-information regularization, and the dual-constrained objective. Use identical models, imagined horizon K, replay data, and environment steps. Measure task return, observation-prediction negative log-likelihood, calibration of latent uncertainty, empirical I_hat, lambda trajectory, and wall-clock/FLOPs. Success is higher return or faster attainment of a fixed return at equal environment interactions, while I_hat tracks H_goal without collapse; a failure signal is lambda saturation together with degraded reconstruction or policy performance.