# Nonexpansive Latent Q-Head

- ID: 70
- Canonical URL: https://synthcore.org/idea/70/nonexpansive-latent-q-head
- API JSON: https://synthcore.org/api/idea/70.json
- API Markdown: https://synthcore.org/api/idea/70.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.22636](https://arxiv.org/abs/2608.22636)
- Category: dynamics
- Solves: stability, sample-efficiency, accuracy
- ML areas: rl, training-dynamics, embedding
- Math tags: dynamical-systems, metric-geometry, approximation-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Represent Q-values using latent coefficients and a convex reconstruction operator rather than an unconstrained linear head. Enforce that reconstruction and compression are sup-norm nonexpansive, so the approximate Bellman operator remains a gamma-contraction and cannot exhibit the usual linear-function-approximation divergence.

## Mathematical statement

The construction uses a latent coefficient field theta in C(L), where (L,rho) is a compact metric space, a reconstruction operator R mapping theta to a Q-function, and a compression operator C mapping Q-functions back to latent coordinates. The required assumptions are ||R theta-R theta'||_infinity <= ||theta-theta'||_infinity and ||C f-C g||_infinity <= ||f-g||_infinity. For the optimal Bellman operator (T*Q)(s,a)=r(s,a)+gamma E[max_{a'} Q(s',a')|s,a], the latent map F(theta)=C T* R theta satisfies ||F(theta)-F(theta')||_infinity <= gamma ||theta-theta'||_infinity, with 0 <= gamma < 1. In an implementation, let L={1,...,M} be inducing points x_j=(s_j,a_j), let p_j(x)>=0 and sum_j p_j(x)=1, and define (R theta)(x)=sum_j p_j(x)theta_j. For a batch B of sampled transitions x_n,r_n,x'_n, define normalized nonnegative compression weights w_{jn} and (C_B y)_j=sum_n w_{jn}y_n, where w_{jn}>=0 and sum_n w_{jn}=1. This compression is nonexpansive in the sup norm. The practical stochastic update is theta_j <- (1-alpha_t)theta_j+alpha_t(C_B y)_j.

## Key formulas

- $$\|R\theta-R\theta'\|_{\infty}\leq \|\theta-\theta'\|_{\infty},\qquad \|Cf-Cg\|_{\infty}\leq \|f-g\|_{\infty}.$$
- $$(T^*Q)(s,a)=r(s,a)+\gamma\,\mathbb{E}\left[\max_{a'}Q(s',a')\mid s,a\right],\qquad F(\theta)=C T^*R\theta.$$
- $$\|F(\theta)-F(\theta')\|_{\infty}\leq \gamma\|\theta-\theta'\|_{\infty},\qquad 0\leq\gamma<1.$$
- $$p_j(x)\geq0,\quad \sum_{j=1}^{M}p_j(x)=1,\quad (R\theta)(x)=\sum_{j=1}^{M}p_j(x)\theta_j;\qquad \theta_{t+1,j}=(1-\alpha_t)\theta_{t,j}+\alpha_t\sum_{n\in B_t}w_{jn}^{(t)}y_n^{(t)}.$$

## Implementation notes

Integrate this at the Q-value output layer, preferably after a frozen CNN, transformer, or MLP state encoder. First collect M inducing state-action points x_j=(s_j,a_j), either by reservoir sampling replay states and enumerating actions or by k-means in the frozen embedding space. The trainable parameter is theta in R^M, not an unconstrained final linear layer. Compute nonnegative partition weights p_j(x)=softmax_j(-d(z(x),z(x_j))/tau), where z is the frozen feature and d is squared Euclidean distance; this implements the reconstruction Q_theta(x)=sum_j p_j(x)theta_j. For each replay minibatch transition (x_n,r_n,x'_n), calculate y_n=r_n+gamma max_a Q_theta_target(x'_n,a), optionally using a slowly updated target copy. Form compression weights w_jn by a Gaussian or top-k kernel around inducing point j, then normalize over n: w_jn=k(z(x_n),z(x_j))/(sum_m k(z(x_m),z(x_j))+epsilon). Update every latent coordinate with theta_j=(1-alpha)theta_j+alpha sum_n w_jn y_n. Clip rewards or theta only for numerical protection, not to claim the contraction. The paper’s mathematical guarantee comes from positivity and normalization of p and w; rho or the distance kernel is an engineering choice and can be learned or replaced by cosine similarity. Estimate approximation error by evaluating Bellman residuals on a held-out replay set, and estimate stability by tracking max_j|theta_j| and Q sup-norm changes. Start with CartPole or a small Atari subset using a frozen 64-dimensional encoder, M=256 inducing points, and compare against a standard linear Q-head and a two-layer MLP head at equal replay and FLOPs. Success means substantially fewer exploding-Q runs, bounded parameter trajectories, and equal-or-better return or Bellman error, especially under off-policy replay and larger learning rates.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a NumPy MVP of simplex-kernel reconstruction, batch-normalized nonnegative compression, latent Bellman updates, and a conventional linear Q-head baseline. The core finite-dimensional contraction check passed: row-sum errors were about 2.2e-16, no violations occurred in 5,000 trials, and the largest observed ratio was 0.708 versus gamma=0.93. In the five-seed matched MDP experiment, the latent head slightly improved mean Bellman residual (2.236 vs 2.378) and Q error (14.767 vs 15.402), but produced much larger Q magnitudes (3.627 vs 1.207) and substantially worse mean policy return (-0.274 vs 0.644); therefore the promised practical win was not observed.

### Mechanism check

- Verdict: Built a NumPy MVP of simplex-kernel reconstruction, batch-normalized nonnegative compression, latent Bellman updates, and a conventional linear Q-head baseline. The core finite-dimensional contraction check passed: row-sum errors were about 2.2e-16, no violations occurred in 5,000 trials, and the largest observed ratio was 0.708 versus gamma=0.93. In the five-seed matched MDP experiment, the latent head slightly improved mean Bellman residual (2.236 vs 2.378) and Q error (14.767 vs 15.402), but produced much larger Q magnitudes (3.627 vs 1.207) and substantially worse mean policy return (-0.274 vs 0.644); therefore the promised practical win was not observed.
- Confidence: 9/10
- Limitations: This is a tiny deterministic synthetic MDP, not CartPole or a neural frozen encoder; the baseline and latent model use different parameterizations and were not FLOP-matched in a full optimizer comparison. No target network, replay-distribution study, reward clipping ablation, larger inducing-point sweep, or statistically powered RL evaluation was tested.

## Artifacts

- [experiment.py](https://synthcore.org/code/37/experiment.py)
- [multiseed_results.json](https://synthcore.org/code/37/multiseed_results.json)
- [report.md](https://synthcore.org/code/37/report.md)
- [results.json](https://synthcore.org/code/37/results.json)
- [Download all files as ZIP](https://synthcore.org/download/37)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.