# Noncommutative controllability regularizer

- ID: 1687
- Canonical URL: https://synthcore.org/idea/1687/noncommutative-controllability-regularizer
- API JSON: https://synthcore.org/api/idea/1687.json
- API Markdown: https://synthcore.org/api/idea/1687.md
- Verification status: unverified
- Source: [arXiv:2608.01836](https://arxiv.org/abs/2608.01836)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: rnn, ssm, training-dynamics, regularization
- Math tags: control-theory, linear-algebra, dynamical-systems, stochastic-processes
- Ratings: usefulness 6/10; difficulty 5/10; novelty 8/10

## Idea description

Equip a recurrent or state-space layer with multiple noncommuting transition operators and regularize the span of finite operator words applied to the input injection matrix. This discourages hidden directions that cannot be reached from the input and may improve long-range input influence, gradient propagation, and robustness under operator switching.

## Mathematical statement

The paper studies the system $dy=[-d\Delta y+A_{1}y+A_{2}Y+B\mathbf{1}_{G}v]dt+YdW_t$, where $A_{1},A_{2}\in\mathbb{R}^{n\times n}$ are coupling matrices and $B\in\mathbb{R}^{n\times m}$ is the control matrix. Its main result states that null controllability is equivalent to a noncommutative word-rank condition involving products of $A_{1}$ and $A_{2}$ applied to $B$. For a neural layer, let $\mathcal{A}=\{A_{1},A_{2}\}$ and define the words of length at most $K$ by $\mathcal{W}_{K}=\{A_{i_r}\cdots A_{i_1}:0\le r\le K,\ i_j\in\{1,2\}\}$. The finite word-controllability matrix is $\mathcal{C}_{K}=[WB]_{W\in\mathcal{W}_{K}}$. If $\operatorname{rank}(\mathcal{C}_{K})<n$, there is an unreachable hidden direction $q\neq0$ satisfying $q^{\top}WB=0$ for every word. A differentiable surrogate is the weighted word Gramian $G_K=\sum_{r=0}^{K}\rho^r\sum_{|W|=r}WBB^{\top}W^{\top}$, where $0<\rho\le1$ discounts long words. Penalizing $-\log\det(G_K+\varepsilon I_n)$ raises all reachable eigenvalues and approximates full-rank controllability.

## Key formulas

- $$dy=\bigl[-d\Delta y+A_{1}y+A_{2}Y+B\mathbf{1}_{G}v\bigr]dt+Y\,dW_t,$$
- $$\mathcal{W}_{K}=\{A_{i_r}\cdots A_{i_1}:0\le r\le K,\ i_j\in\{1,2\}\},\qquad \mathcal{C}_{K}=\bigl[WB\bigr]_{W\in\mathcal{W}_{K}},$$
- $$\operatorname{rank}(\mathcal{C}_{K})<n\ \Longrightarrow\ \exists q\ne0:\ q^{\top}WB=0\quad\forall W\in\mathcal{W}_{K},$$
- $$G_K=\sum_{r=0}^{K}\rho^r\sum_{\substack{W\in\mathcal{W}_{K}\\ |W|=r}}WB B^{\top}W^{\top},\qquad \mathcal{L}_{\mathrm{reach}}=-\log\det(G_K+\varepsilon I_n).$$

## Implementation notes

Integrate the method into a gated RNN or linear state-space block. Use hidden state $h_t\in\mathbb{R}^{n}$, input projection $B\in\mathbb{R}^{n\times m}$, and two transition operators $A_1,A_2\in\mathbb{R}^{n\times n}$, for example $h_{t+1}=\phi(A_1h_t+g_tA_2h_t+Bu_t)$ with a scalar or vector gate $g_t$. Add $\lambda\mathcal{L}_{reach}$ to the task loss.

Pseudocode: set a list of word blocks $V=[B]$ and initialize $G=BB^\top$. For $r=1,\ldots,K$, create a new list by computing $X_1=A_1X$ and $X_2=A_2X$ for every $X$ in the previous list; optionally normalize each $X$ by $\max(\|X\|_F,\delta)$ to prevent exploding products. Accumulate $G\leftarrow G+\rho^rXX^\top$ for each new block $X$, then compute $\mathcal{L}_{reach}=-\log\det(G+\varepsilon I)$. Use a Cholesky-based log determinant for numerical stability. The exact mathematical quantities are the operator words, their weighted Gramian, and its eigenvalues; $K$, $\rho$, $\varepsilon$, and $\lambda$ must be selected empirically. Track the smallest eigenvalue, effective rank, and gradient norms across time. Begin with a 64-unit RNN on copy-memory and permuted-sequential-MNIST, comparing a standard RNN, the same architecture with only $BB^\top$, and the proposed word penalty. Test $K\in\{2,3,4\}$. Success means higher long-context accuracy, less gradient decay, and a larger minimum Gramian eigenvalue at equal parameter count and FLOPs.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.