Hessian-guided orthogonal sparsification

Integrate the method into a square orthogonal transform used for channel mixing, an MLP feature rotation, or a token-mixing layer. Represent the transform as Q = Cayley(S), with S skew-symmetric, or as a product of Givens rotations; both representations preserve Q^T Q = I. Add the concentration loss L_conc = -lambda f_r(Q) / r to the task loss, initially with a small lambda and an annealing schedule. Every K optimizer steps, compute the tangent-gradient norm of the concentration objective. If the norm is small but f_r(Q) / r is still below a target, enumerate row pairs a < b and compute H_ab directly from the displayed formula. Compute the same values on Q^T for column pairs. If the largest curvature exceeds a threshold tau, choose that pair and apply Q <- exp(eta A_ab) Q. Use backtracking over eta in {eta0, eta0/2, eta0/4, ...} and accept only if f_r(Q) increases while orthogonality error remains below 1e-5. Then resume gradient training. The curvature and objective are computed exactly from Q; tau, K, lambda, and the pruning threshold are empirical hyperparameters. First test a 2-layer CIFAR-10 MLP with width 128 and one orthogonal channel mixer. Compare dense orthogonal training, concentration regularization without escape steps, and the proposed method. Measure validation accuracy, f_r(Q) / r, number of entries above 1e-3, orthogonality error, training loss at equal FLOPs, and latency after replacing Q by the signed permutation obtained from each row's largest-magnitude entry with collision resolution. Success means faster concentration and near-baseline accuracy after hard sparsification.