Lie-Bracket Orthogonal Mixer

Integrate this at the channel-mixing operation of an MLP or recurrent block. Let the feature width be \(d\), store two sparse trainable matrices \(A,B\in\mathbb{R}^{d\times d}\), and enforce skew symmetry after every optimizer step with \(A\leftarrow(A-A^{\mathsf T})/2\) and \(B\leftarrow(B-B^{\mathsf T})/2\). Construct \(C_0=B\), then iterate \(C_{k+1}=AC_k-C_kA\) for \(k=0,\ldots,K-1\). Normalize each direction by \(C_k\leftarrow C_k/(\|C_k\|_F+10^{-6})\), using a separate learned scalar for scale. For each input token or hidden vector \(h\), compute the ordered product \(Q=\exp(\theta_0C_0)\exp(\theta_1C_1)\cdots\exp(\theta_KC_K)\) and output \(Qh\), followed by the usual nonlinear MLP sublayer and residual connection. The \(\theta_k\) may be global learned scalars, per-layer scalars, or generated from a pooled input using a tiny gating MLP. Use exact matrix exponentials for \(d\leq64\); for larger widths use scaling-and-squaring or a truncated exponential with periodic orthogonality checks. The paper supplies the skew-symmetric Lie algebra, adjoint recursion, and commutator propagation principle; normalization, coefficient parameterization, and truncation are engineering choices. First test on a two-layer MLP and a GRU-like sequence model on CIFAR-10 or sequential MNIST, comparing against a dense mixer with equal parameter count and against a standard orthogonal parametrization. Measure training loss at equal FLOPs, hidden-state norm drift, gradient norms, and final accuracy. Success means comparable or better accuracy with fewer mixer parameters and bounded activation norms; in the recurrent model, also test whether long-horizon gradient decay is reduced.