# Zero-mode MPS front-end for constrained sequence models

- ID: 2959
- Canonical URL: https://synthcore.org/idea/2959/zero-mode-mps-front-end-for-constrained-sequence-models
- API JSON: https://synthcore.org/api/idea/2959.json
- API Markdown: https://synthcore.org/api/idea/2959.md
- Verification status: unverified
- Source: [arXiv:2608.31165](https://arxiv.org/abs/2608.31165)
- Category: architecture
- Solves: stability, sample-efficiency, accuracy
- ML areas: transformer, embedding, data-augmentation
- Math tags: linear-algebra, tensor-decomposition, algebra, dynamical-systems
- Ratings: usefulness 5/10; difficulty 6/10; novelty 6/10

## Idea description

Build a constrained autoregressive model whose initial logits are generated from a translation-invariant MPS associated with a local zero-mode construction. The MPS supplies a structured valid distribution before a Transformer residual is added, so the model starts on the constraint manifold instead of learning validity through a penalty.

## Mathematical statement

The paper represents the constrained Hamiltonian as $H=\mathcal{P}(\sum_j\mathbf{h}_j)\mathcal{P}$, where $\mathbf{h}$ is the single-site unconstrained Hamiltonian, $j$ indexes sites, and $\mathcal{P}$ projects onto globally allowed configurations. Its construction selects local MPS tensors from eigenspaces of the local operator. The extracted example contains $M^{L}_{1}=M^{R}_{1}=M^{U}_{1}=\frac{1}{\sqrt{6}}\begin{pmatrix}0&0\\1&1\end{pmatrix}$ and $M_{1}^{O}=\frac{\pm1}{\sqrt{2}}\begin{pmatrix}0&0\\1&1\end{pmatrix}$. For a sequence $s=(s_1,\ldots,s_n)$, let $M_a\in\mathbb{R}^{D\times D}$ be the tensor for token $a$, and define the open-boundary MPS amplitude $\psi_M(s)=u^\top M_{s_1}\cdots M_{s_n}v$, where $u,v\in\mathbb{R}^{D}$ are boundary vectors. The adapted base distribution is $p_0(s)=\mathcal{P}(s)|\psi_M(s)|^2/Z$, with $Z=\sum_{s'}\mathcal{P}(s')|\psi_M(s')|^2$. The Transformer learns a residual score $r_\theta(s_{<t},a)$ on top of the MPS conditional distribution. The mechanism is exact support restriction through $\mathcal{P}$ plus a low-bond-dimension tensor prior whose amplitudes are computed by matrix products.

## Key formulas

- $$H=\mathcal{P}\left(\sum_j\mathbf{h}_j\right)\mathcal{P}$$
- $$M^{L}_{1}=M^{R}_{1}=M^{U}_{1}=\frac{1}{\sqrt{6}}\begin{pmatrix}0&0\\1&1\end{pmatrix},\qquad M_{1}^{O}=\frac{\pm 1}{\sqrt{2}}\begin{pmatrix}0&0\\1&1\end{pmatrix}$$
- $$\psi_M(s_1,\ldots,s_n)=u^\top M_{s_1}M_{s_2}\cdots M_{s_n}v,\qquad p_0(s)=\frac{\mathcal{P}(s)|\psi_M(s)|^2}{\sum_{s'}\mathcal{P}(s')|\psi_M(s')|^2}$$
- $$\ell_\theta(a\mid s_{<t})=\log p_0(a\mid s_{<t})+r_\theta(s_{<t},a)-\log\sum_b p_0(b\mid s_{<t})e^{r_\theta(s_{<t},b)}$$

## Implementation notes

(1) Integration point: use an autoregressive Transformer for sequences obeying a known local rule, such as no adjacent 1s, finite-range exclusion, or legal symbolic actions. Add an MPS base distribution immediately before the final token softmax. Store one matrix $M_a\in\mathbb{R}^{D\times D}$ per token, start with bond dimension $D=2$, and maintain the prefix product $Q_t=M_{s_1}\cdots M_{s_{t-1}}$. For each candidate token $a$, compute its base score from the MPS contraction and set the score to $-\infty$ if appending $a$ violates the constraint. (2) Pseudocode: initialize $Q=I$; at position $t$, enumerate legal tokens; calculate $b_a=\log(|u^\top QM_av|^2+\epsilon)$; calculate Transformer residual $r_a=r_\theta(h_t,a)$; form $z_a=b_a+r_a$; normalize with log-sum-exp; train with teacher forcing or sample from the normalized distribution; then update $Q\leftarrow QM_{s_t}$. Use the supplied matrices as an initialization when token labels correspond to the local eigenbasis, and learn additional matrices for unmatched tokens. (3) Compute matrix products and legality masks exactly; optionally fine-tune $M_a,u,v$ with a small learning rate, while keeping the validity mask fixed. (4) First experiment: train a two-layer, four-head Transformer on a synthetic length-64 exclusion language and compare against the same model with a learned output head and against a model using only a hard validity mask. Match parameter count, optimizer, training steps, and processed tokens. (5) Pre-register mechanism tests: invalid generated sequences must remain below $10^{-6}$; at 10%, 25%, and 50% of baseline training tokens, the MPS model should achieve at least 5% lower validation NLL; and removing the MPS prior while retaining the hard mask should reduce loss improvement per token by at least 10% during the first 20% of training. Record NLL, invalid rate, gradient norm, and loss-versus-token curves. The transfer is falsified if the MPS prior gives no early-training improvement after parameter and FLOP matching, or if all gains disappear when the residual Transformer is sufficiently trained.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.