{
 "artifacts": null,
 "category": "geometry",
 "description": "Parameterize an orthogonal or semi-orthogonal neural weight matrix directly on the Stiefel manifold and update it with a Cayley retraction instead of unconstrained SGD plus a penalty or QR projection. The update preserves orthogonality exactly, is second-order accurate for the appropriate metric, and avoids the cubic QR factorization at every optimizer step.",
 "formulas_latex": [
  "$$\\operatorname{Cay}(A)=(I-A)^{-1}(I+A),\\qquad A^T=-A,$$",
  "$$W=\\xi X^T-X\\xi^T-X(X^T\\xi)X^T,\\qquad W^T=-W,\\qquad WX=\\xi,$$",
  "$$R_X^{\\mathrm{Cay}}(\\eta\\xi)=\\left(I-\\frac{\\eta W}{2}\\right)^{-1}\\left(I+\\frac{\\eta W}{2}\\right)X,$$",
  "$$\\|\\mathcal{X}\\|_2=\\max\\left\\{\\frac{\\sigma_i}{\\sqrt{1+\\sigma_i^2}},\\;\\frac{\\sqrt{2}\\sqrt{\\sigma_i^2+\\sigma_j^2}}{\\sqrt{1+\\sigma_i^2}+\\sqrt{1+\\sigma_j^2}}\\right\\}\\leq 1,$$"
 ],
 "id": 2931,
 "implementation": "Integrate this at the optimizer update for a square orthogonal weight X in an MLP, residual mixer, or orthogonal adapter. Store X with shape n x p and initialize it by QR(X_random). After backpropagation, let G be the Euclidean gradient and form a tangent direction xi = G - X sym(X^T G), where sym(A)=(A+A^T)/2. Use the tangent-compatible generator W = xi X^T - X xi^T - X(X^T xi)X^T. Apply the optimizer's scalar step size eta, form A=eta W/2, and compute the new parameter by solving (I-A)Y=(I+A)X; set X \u003c- Y. Do not explicitly form a matrix inverse. If the estimated spectral norm ||A||_2 exceeds 0.5, repeatedly halve eta until it is below 0.5, creating a conservative local-inverse trust region. Estimate ||A||_2 with 3-5 power iterations. For the paper's conditioning diagnostic, compute or approximate the singular values sigma_i of the relevant tangent/operator blocks and evaluate the displayed maximum; log this value and the update norm. The first experiment should use a 2-layer width-512 MLP on CIFAR-10 with one orthogonal hidden mixer. Compare unconstrained SGD, Euclidean SGD plus ||X^T X-I||_F^2, QR projection after every step, and Cayley updates, matching parameter count, optimizer, steps, and measured FLOPs. Record accuracy, wall-clock time, orthogonality error, gradient norm, loss spikes, and conditioning estimates over five seeds. Pre-register that Cayley maintains ||X^T X-I||_F below 1e-5 without QR, has at least 20% lower update overhead than QR, and shows fewer instability events while the conditioning proxy is below 0.9. Deliberately violate the trust-region threshold as an ablation; the proposed mechanism predicts increased loss spikes or degraded conditioning. The transfer is falsified if Cayley is not faster than QR at equal accuracy, orthogonality error exceeds 1e-4, or conditioning crossings fail to predict loss spikes better than a shuffled-threshold baseline.",
 "math_summary": "For a Stiefel point X in R^{n x p}, X^T X=I_p, construct a skew-symmetric generator W from a tangent direction xi by W = xi X^T - X xi^T - X(X^T xi)X^T; then W^T=-W and WX=xi. The Cayley retraction is R_X(xi) = (I - W/2)^{-1}(I + W/2)X, with an optional scalar step size eta replacing W by eta W. Since eta W is skew-symmetric, Q=(I-eta W/2)^{-1}(I+eta W/2) is orthogonal, so R_X(eta xi)^T R_X(eta xi)=I_p. The paper's conditioning calculation bounds the tangent-coordinate operator by the largest of |delta_i| = sigma_i/sqrt(1+sigma_i^2) and gamma_ij sqrt(2)sqrt(sigma_i^2+sigma_j^2) = sqrt(2)sqrt(sigma_i^2+sigma_j^2)/(sqrt(1+sigma_i^2)+sqrt(1+sigma_j^2)); here sigma_i are the singular values appearing in the operator's block decomposition, delta_i and gamma_ij are its diagonal and off-diagonal block gains. The diagonal gains are at most one, and the off-diagonal expression is also bounded, giving a practical spectral indicator for whether the retraction map is well-conditioned. The inverse is only local, so updates should enforce ||eta W||_2 below a fixed trust-region threshold.",
 "math_tags": [
  "geometry",
  "linear-algebra",
  "numerical-analysis",
  "dynamical-systems"
 ],
 "ml_areas": [
  "optimizer",
  "mlp",
  "initialization",
  "fine-tuning"
 ],
 "paper": {
  "arxiv_id": "2608.28073",
  "arxiv_url": "https://arxiv.org/abs/2608.28073",
  "summary_what_math_gives_to_ml": "The paper gives explicit Cayley and polar-light retractions on the Stiefel manifold together with conditioning bounds for their coordinate maps and local inverses. This suggests replacing penalty-based orthogonality or repeated QR projections in neural layers with an exactly orthogonality-preserving Cayley update whose conditioning can be monitored from singular values of the tangent-coordinate operator. The most direct transfer is an orthogonal linear layer or adapter trained on the Stiefel manifold, where the retraction provides a cheap second-order update and the paper's spectral bound supplies a falsifiable stability diagnostic.",
  "title": "Conditioning and interpolation error bounds for second-order Stiefel retractions with closed-form inverses",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 5,
  "usefulness": 6
 },
 "solves": [
  "stability",
  "accuracy",
  "speedup"
 ],
 "title": "Conditioned Cayley updates for orthogonal neural layers",
 "url": "https://synthcore.org/idea/2931/conditioned-cayley-updates-for-orthogonal-neural-layers",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}