Nonexpansive Latent Q-Head

Integrate this at the Q-value output layer, preferably after a frozen CNN, transformer, or MLP state encoder. First collect M inducing state-action points x_j=(s_j,a_j), either by reservoir sampling replay states and enumerating actions or by k-means in the frozen embedding space. The trainable parameter is theta in R^M, not an unconstrained final linear layer. Compute nonnegative partition weights p_j(x)=softmax_j(-d(z(x),z(x_j))/tau), where z is the frozen feature and d is squared Euclidean distance; this implements the reconstruction Q_theta(x)=sum_j p_j(x)theta_j. For each replay minibatch transition (x_n,r_n,x'_n), calculate y_n=r_n+gamma max_a Q_theta_target(x'_n,a), optionally using a slowly updated target copy. Form compression weights w_jn by a Gaussian or top-k kernel around inducing point j, then normalize over n: w_jn=k(z(x_n),z(x_j))/(sum_m k(z(x_m),z(x_j))+epsilon). Update every latent coordinate with theta_j=(1-alpha)theta_j+alpha sum_n w_jn y_n. Clip rewards or theta only for numerical protection, not to claim the contraction. The paper’s mathematical guarantee comes from positivity and normalization of p and w; rho or the distance kernel is an engineering choice and can be learned or replaced by cosine similarity. Estimate approximation error by evaluating Bellman residuals on a held-out replay set, and estimate stability by tracking max_j|theta_j| and Q sup-norm changes. Start with CartPole or a small Atari subset using a frozen 64-dimensional encoder, M=256 inducing points, and compare against a standard linear Q-head and a two-layer MLP head at equal replay and FLOPs. Success means substantially fewer exploding-Q runs, bounded parameter trajectories, and equal-or-better return or Bellman error, especially under off-policy replay and larger learning rates.