# Lie-Bracket Orthogonal Mixer

- ID: 2602
- Canonical URL: https://synthcore.org/idea/2602/lie-bracket-orthogonal-mixer
- API JSON: https://synthcore.org/api/idea/2602.json
- API Markdown: https://synthcore.org/api/idea/2602.md
- Verification status: unverified
- Source: [arXiv:2608.20094](https://arxiv.org/abs/2608.20094)
- Category: architecture
- Solves: stability, accuracy, memory
- ML areas: mlp, rnn, embedding
- Math tags: lie-groups, algebra, linear-algebra, dynamical-systems
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Replace a dense unconstrained channel-mixing matrix with a differentiable product of exponentials of a few skew-symmetric generators and their iterated commutators. The resulting layer is exactly orthogonal, preserves feature norms, and can express rotations in directions not explicitly stored as independent parameters. This is especially suitable for residual MLP blocks, recurrent state transitions, and networks processing rotation- or pose-valued features.

## Mathematical statement

The paper defines the Lie algebra of rotations as \(\mathfrak{so}(n)=\{X\in\mathbb{R}^{n\times n}:X^{\mathsf T}=-X\}\), with matrix Lie group \(\operatorname{SO}(n)=\{Q:Q^{\mathsf T}Q=I,\det Q=1\}\). For a primary generator \(A\in\mathfrak{so}(n)\) and a localized control direction \(B\in\mathfrak{so}(n)\), it defines the adjoint operator \(\operatorname{ad}_A(C)=[A,C]\) and the generated family \(C_k=\operatorname{ad}_A^k(B)\), where \(C_0=B\) and \(C_{k+1}=[A,C_k]\). Every \(C_k\) remains skew-symmetric, so \(\exp(tC_k)\in\operatorname{SO}(n)\). The implementable commutator synthesis identity is \(\exp(\varepsilon A)\exp(\varepsilon B)\exp(-\varepsilon A)\exp(-\varepsilon B)=\exp(\varepsilon^2[A,B]+O(\varepsilon^3))\) for small \(\varepsilon\). The network uses these generated skew directions as a structured basis for an orthogonal feature mixer, with trainable scalar coefficients and optionally trainable sparse entries of \(A\) and \(B\).

## Key formulas

- $$\operatorname{ad}_{A}:\mathfrak{so}(4)\to\mathfrak{so}(4),\qquad C\mapsto[A,C].$$
- $$C_0=B,\qquad C_{k+1}=[A,C_k],\qquad C_k=\operatorname{ad}_{A}^{k}(B).$$
- $$\exp(\varepsilon A)\exp(\varepsilon B)\exp(-\varepsilon A)\exp(-\varepsilon B)=\exp\!\left(\varepsilon^2[A,B]+O(\varepsilon^3)\right).$$
- $$h_{\mathrm{out}}=\left(\prod_{k=0}^{K}\exp(\theta_k C_k)\right)h_{\mathrm{in}},\qquad A^{\mathsf T}=-A,\ B^{\mathsf T}=-B.$$

## Implementation notes

Integrate this at the channel-mixing operation of an MLP or recurrent block. Let the feature width be \(d\), store two sparse trainable matrices \(A,B\in\mathbb{R}^{d\times d}\), and enforce skew symmetry after every optimizer step with \(A\leftarrow(A-A^{\mathsf T})/2\) and \(B\leftarrow(B-B^{\mathsf T})/2\). Construct \(C_0=B\), then iterate \(C_{k+1}=AC_k-C_kA\) for \(k=0,\ldots,K-1\). Normalize each direction by \(C_k\leftarrow C_k/(\|C_k\|_F+10^{-6})\), using a separate learned scalar for scale. For each input token or hidden vector \(h\), compute the ordered product \(Q=\exp(\theta_0C_0)\exp(\theta_1C_1)\cdots\exp(\theta_KC_K)\) and output \(Qh\), followed by the usual nonlinear MLP sublayer and residual connection. The \(\theta_k\) may be global learned scalars, per-layer scalars, or generated from a pooled input using a tiny gating MLP. Use exact matrix exponentials for \(d\leq64\); for larger widths use scaling-and-squaring or a truncated exponential with periodic orthogonality checks. The paper supplies the skew-symmetric Lie algebra, adjoint recursion, and commutator propagation principle; normalization, coefficient parameterization, and truncation are engineering choices. First test on a two-layer MLP and a GRU-like sequence model on CIFAR-10 or sequential MNIST, comparing against a dense mixer with equal parameter count and against a standard orthogonal parametrization. Measure training loss at equal FLOPs, hidden-state norm drift, gradient norms, and final accuracy. Success means comparable or better accuracy with fewer mixer parameters and bounded activation norms; in the recurrent model, also test whether long-horizon gradient decay is reduced.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.