# Controllability-Rank Regularizer

- ID: 2603
- Canonical URL: https://synthcore.org/idea/2603/controllability-rank-regularizer
- API JSON: https://synthcore.org/api/idea/2603.json
- API Markdown: https://synthcore.org/api/idea/2603.md
- Verification status: unverified
- Source: [arXiv:2608.20094](https://arxiv.org/abs/2608.20094)
- Category: regularization
- Solves: accuracy, stability, generalization
- ML areas: mlp, transformer, regularization
- Math tags: lie-groups, algebra, linear-algebra, control-theory
- Ratings: usefulness 5/10; difficulty 4/10; novelty 7/10

## Idea description

Regularize learned skew generators so that their iterated Lie brackets span many independent feature-mixing directions rather than collapsing to commuting or redundant matrices. This turns the paper's controllability family into a differentiable diversity objective for structured neural layers.

## Mathematical statement

For \(A,B\in\mathfrak{so}(d)\), the generated directions are \(C_k=\operatorname{ad}_A^k(B)\), with \(C_0=B\) and \(C_{k+1}=AC_k-C_kA\). Equip matrices with the Frobenius inner product \(\langle X,Y\rangle_F=\operatorname{tr}(X^{\mathsf T}Y)\). Form the controllability Gram matrix \(G\in\mathbb{R}^{(K+1)\times(K+1)}\), \(G_{ij}=\langle C_i,C_j\rangle_F\). If the directions are linearly independent, \(G\) has larger rank and positive smallest eigenvalue; if the brackets collapse, \(G\) is singular or ill-conditioned. A differentiable regularizer is \(\mathcal{L}_{\mathrm{rank}}=-\log\det(G+\delta I)\), optionally after normalizing \(\widetilde C_k=C_k/(\|C_k\|_F+\epsilon)\). Here \(K\) is the number of generated brackets, \(\delta>0\) stabilizes the determinant, and \(I\) is the identity matrix. This is a neural adaptation of the paper's finite-dimensional propagation criterion; maximizing rank alone is not asserted to guarantee task performance.

## Key formulas

- $$C_0=B,\qquad C_{k+1}=\operatorname{ad}_A(C_k)=[A,C_k]=AC_k-C_kA.$$
- $$G_{ij}=\langle \widetilde C_i,\widetilde C_j\rangle_F=\operatorname{tr}(\widetilde C_i^{\mathsf T}\widetilde C_j),\qquad \widetilde C_k=\frac{C_k}{\|C_k\|_F+\epsilon}.$$
- $$\mathcal{L}_{\mathrm{rank}}=-\log\det(G+\delta I).$$

## Implementation notes

Attach the regularizer to a structured orthogonal mixer with trainable skew-symmetric matrices \(A\) and \(B\). During each training step, compute \(C_0=B\) and recursively form \(C_1,\ldots,C_K\) using matrix commutators. Normalize each \(C_k\) by its Frobenius norm, construct the small \((K+1)\times(K+1)\) Gram matrix \(G\), and add \(\lambda\mathcal{L}_{\mathrm{rank}}\) to the task loss. Use \(K=2\) or \(3\) initially so the extra cost is modest; evaluate the log determinant with a Cholesky factorization of \(G+\delta I\), using \(\delta=10^{-4}\). Stop gradients through normalization only if exploding gradients occur. The exact mathematical computation is the iterated adjoint family and its Gram determinant; \(\lambda\), \(K\), normalization, and whether to apply the term only during warm-up are empirical choices. Run a cheap ablation on a CIFAR-10 MLP, a small transformer feed-forward mixer, or sequential MNIST: compare no regularizer, orthogonality-only, and the Lie-rank penalty at matched parameter count. Log the singular values of the flattened family \([\operatorname{vec}(C_0),\ldots,\operatorname{vec}(C_K)]\), task accuracy, gradient variance, and loss across random seeds. The expected signal is a larger minimum singular value and less generator collapse, followed by improved validation accuracy or robustness under hidden-width reduction. Reject the method if the determinant increases without task benefit or if its overhead exceeds the structured layer's savings.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.