Contractive Misspecification-Regularized State Model

(1) Integration point: replace the transition component of a discrete-latent RNN or compact state-space model with a row-stochastic student matrix $\widehat K_\theta\in\mathbb{R}^{m\times m}$, parameterized by row-wise softmax logits. Keep the teacher transition matrix $K$ frozen, obtained from a larger trained model, an HMM baseline, or empirical transition counts. Use the usual observation or emission update to produce posteriors, but add the robustness penalty to the sequence loss.

(2) Pseudocode:

for batch of sequences:
    Khat = row_softmax(student_logits)
    C = 0
    eps = 0
    for i in range(m):
        eps = max(eps, 0.5 * sum(abs(K[i] - Khat[i])))
        for j in range(m):
            C = max(C, 0.5 * sum(abs(Khat[i] - Khat[j])))
    Lrobust = lambda_eps * eps + lambda_C * relu(C - c_target)**2
    posterior = initial_posterior
    for t in range(T):
        predictive = posterior @ Khat
        posterior = normalize(predictive * emission_probability(y[t]))
        accumulate_task_or_distillation_loss(posterior)
    loss = sequence_loss + Lrobust
    backpropagate(loss)

The paper's contraction coefficient is C, and its model discrepancy is eps; the geometric bound predicts the long-horizon posterior error as eps/(1-C).

(3) Compute $\mathsf{C}(\widehat K)$ exactly for small latent state spaces. Compute $\widehat\epsilon$ exactly when teacher and student share states. If the teacher has continuous states, estimate both quantities on a minibatch of representative states using sampled next-state distributions; use the maximum over sampled pairs as a conservative proxy. The mathematical guarantee assumes a uniform bound, so monitor the empirical maximum rather than only the mean discrepancy. Clip or parameterize the target so that $c_{\mathrm{target}}\leq0.95$.

(4) First experiment: train a 16- or 32-state latent HMM-RNN student to imitate a 128-state teacher on sequential MNIST, permuted MNIST, or a synthetic switching linear dynamical system. Compare ordinary distillation, mismatch penalty only, contraction penalty only, and the combined objective at equal parameter count. Evaluate one-step likelihood, posterior TV distance to the teacher, and degradation after sequence lengths 2x, 4x, and 8x longer than training sequences. Success is a lower long-horizon posterior error and a measured error floor close to the predicted $\widehat\epsilon/(1-\mathsf{C})$, without sacrificing short-horizon likelihood.