# Conditioned Cayley updates for orthogonal neural layers

- ID: 2931
- Canonical URL: https://synthcore.org/idea/2931/conditioned-cayley-updates-for-orthogonal-neural-layers
- API JSON: https://synthcore.org/api/idea/2931.json
- API Markdown: https://synthcore.org/api/idea/2931.md
- Verification status: unverified
- Source: [arXiv:2608.28073](https://arxiv.org/abs/2608.28073)
- Category: geometry
- Solves: stability, accuracy, speedup
- ML areas: optimizer, mlp, initialization, fine-tuning
- Math tags: geometry, linear-algebra, numerical-analysis, dynamical-systems
- Ratings: usefulness 6/10; difficulty 5/10; novelty 5/10

## Idea description

Parameterize an orthogonal or semi-orthogonal neural weight matrix directly on the Stiefel manifold and update it with a Cayley retraction instead of unconstrained SGD plus a penalty or QR projection. The update preserves orthogonality exactly, is second-order accurate for the appropriate metric, and avoids the cubic QR factorization at every optimizer step.

## Mathematical statement

For a Stiefel point X in R^{n x p}, X^T X=I_p, construct a skew-symmetric generator W from a tangent direction xi by W = xi X^T - X xi^T - X(X^T xi)X^T; then W^T=-W and WX=xi. The Cayley retraction is R_X(xi) = (I - W/2)^{-1}(I + W/2)X, with an optional scalar step size eta replacing W by eta W. Since eta W is skew-symmetric, Q=(I-eta W/2)^{-1}(I+eta W/2) is orthogonal, so R_X(eta xi)^T R_X(eta xi)=I_p. The paper's conditioning calculation bounds the tangent-coordinate operator by the largest of |delta_i| = sigma_i/sqrt(1+sigma_i^2) and gamma_ij sqrt(2)sqrt(sigma_i^2+sigma_j^2) = sqrt(2)sqrt(sigma_i^2+sigma_j^2)/(sqrt(1+sigma_i^2)+sqrt(1+sigma_j^2)); here sigma_i are the singular values appearing in the operator's block decomposition, delta_i and gamma_ij are its diagonal and off-diagonal block gains. The diagonal gains are at most one, and the off-diagonal expression is also bounded, giving a practical spectral indicator for whether the retraction map is well-conditioned. The inverse is only local, so updates should enforce ||eta W||_2 below a fixed trust-region threshold.

## Key formulas

- $$\operatorname{Cay}(A)=(I-A)^{-1}(I+A),\qquad A^T=-A,$$
- $$W=\xi X^T-X\xi^T-X(X^T\xi)X^T,\qquad W^T=-W,\qquad WX=\xi,$$
- $$R_X^{\mathrm{Cay}}(\eta\xi)=\left(I-\frac{\eta W}{2}\right)^{-1}\left(I+\frac{\eta W}{2}\right)X,$$
- $$\|\mathcal{X}\|_2=\max\left\{\frac{\sigma_i}{\sqrt{1+\sigma_i^2}},\;\frac{\sqrt{2}\sqrt{\sigma_i^2+\sigma_j^2}}{\sqrt{1+\sigma_i^2}+\sqrt{1+\sigma_j^2}}\right\}\leq 1,$$

## Implementation notes

Integrate this at the optimizer update for a square orthogonal weight X in an MLP, residual mixer, or orthogonal adapter. Store X with shape n x p and initialize it by QR(X_random). After backpropagation, let G be the Euclidean gradient and form a tangent direction xi = G - X sym(X^T G), where sym(A)=(A+A^T)/2. Use the tangent-compatible generator W = xi X^T - X xi^T - X(X^T xi)X^T. Apply the optimizer's scalar step size eta, form A=eta W/2, and compute the new parameter by solving (I-A)Y=(I+A)X; set X <- Y. Do not explicitly form a matrix inverse. If the estimated spectral norm ||A||_2 exceeds 0.5, repeatedly halve eta until it is below 0.5, creating a conservative local-inverse trust region. Estimate ||A||_2 with 3-5 power iterations. For the paper's conditioning diagnostic, compute or approximate the singular values sigma_i of the relevant tangent/operator blocks and evaluate the displayed maximum; log this value and the update norm. The first experiment should use a 2-layer width-512 MLP on CIFAR-10 with one orthogonal hidden mixer. Compare unconstrained SGD, Euclidean SGD plus ||X^T X-I||_F^2, QR projection after every step, and Cayley updates, matching parameter count, optimizer, steps, and measured FLOPs. Record accuracy, wall-clock time, orthogonality error, gradient norm, loss spikes, and conditioning estimates over five seeds. Pre-register that Cayley maintains ||X^T X-I||_F below 1e-5 without QR, has at least 20% lower update overhead than QR, and shows fewer instability events while the conditioning proxy is below 0.9. Deliberately violate the trust-region threshold as an ablation; the proposed mechanism predicts increased loss spikes or degraded conditioning. The transfer is falsified if Cayley is not faster than QR at equal accuracy, orthogonality error exceeds 1e-4, or conditioning crossings fail to predict loss spikes better than a shuffled-threshold baseline.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.