# Off-Diagonal Constraint Homotopy for Nontransverse Sparse Weights

- ID: 832
- Canonical URL: https://synthcore.org/idea/832/off-diagonal-constraint-homotopy-for-nontransverse-sparse-we
- API JSON: https://synthcore.org/api/idea/832.json
- API Markdown: https://synthcore.org/api/idea/832.md
- Verification status: unverified
- Source: [arXiv:2607.12877](https://arxiv.org/abs/2607.12877)
- Category: geometry
- Solves: stability, accuracy, generalization
- ML areas: attention, ssm, optimizer, training-dynamics
- Math tags: optimization, geometry, differential-geometry, linear-algebra
- Ratings: usefulness 6/10; difficulty 5/10; novelty 8/10

## Idea description

When a chosen sparse support is geometrically incompatible with exact orthogonality, temporarily optimize on a nearby off-diagonally perturbed Stiefel constraint rather than forcing a singular Newton system. Anneal the perturbation to zero after the active support has stabilized, using the paper's O(||Delta||_F) KKT guarantee to control the residual of the original orthogonality-constrained problem.

## Mathematical statement

The paper identifies a failure mode in which the active sparsity manifold M_g and St(n,r) do not intersect transversely: the common normal space N_X St intersect N_X M_g can be nonzero even for points with identical sign and support. Its examples show that support alone does not determine the geometry: one point has dim(N_X St intersect N_X M_g)=0, while another point with the same signed support has dimension 1. The proposed remedy is an off-diagonally perturbed Stiefel family, represented as St_Delta(n,r)={X: X^T X=I_r+Delta}, where Delta is symmetric with zero diagonal. Generic Delta restores the identification geometry, while a solution satisfying the perturbed constraint has an O(||Delta||_F) KKT guarantee for the original problem. Here Delta is a small symmetric r by r matrix, ||Delta||_F is its Frobenius norm, X_Delta is a stationary point of the perturbed problem, and C is a problem-dependent local constant. The resulting homotopy solves better-conditioned perturbed problems, reduces Delta, and finishes with exact orthogonality.

## Key formulas

- $$\dim\bigl(N_X\mathrm{St}\cap N_X\mathcal{M}_g\bigr)=0\quad\text{(clean/transverse case)},\qquad \dim\bigl(N_X\mathrm{St}\cap N_X\mathcal{M}_g\bigr)>0\quad\text{(nontransverse case)},$$
- $$\mathrm{St}_\Delta(n,r):=\{X\in\mathbb{R}^{n\times r}:X^\top X=I_r+\Delta\},\qquad \Delta=\Delta^\top,\quad \Delta_{ii}=0,$$
- $$\|\mathrm{KKT}_{\mathrm{St}}(X_\Delta)\|\le C\|\Delta\|_F,$$
- $$\Delta_{k+1}=\rho\Delta_k,\qquad 0<\rho<1,\qquad \|X_k^\top X_k-I_r-\Delta_k\|_F\le\varepsilon_k.$$

## Implementation notes

(1) Integration point: apply this to a sparse orthogonal projection used in multi-head attention, an orthogonal recurrent/SSM transition, or a factor in a low-memory MLP. The parameter remains a trainable matrix, but its orthogonality projection/retraction is changed from X^T X=I to X^T X=I+Delta. Choose a symmetric zero-diagonal Delta with small random off-diagonal entries, normalized so ||Delta||_F=delta_0. (2) Pseudocode: initialize X orthogonal and set Delta=Delta_0. For each minibatch, compute a sparse proximal-gradient or Newton-CG update while enforcing X^T X approximately equal to I+Delta. A practical retraction is X <- Y(Y^T Y)^(-1/2)(I+Delta)^(1/2) for candidate Y, provided I+Delta is positive definite. Track the support and restricted Newton residual. Once the support has remained fixed for q steps and the Newton solve is stable, reduce Delta <- rho Delta and repeat. At each stage require the perturbed feasibility tolerance and a loss decrease; otherwise retain the previous Delta and use a damped gradient step. Finish with Delta=0 and several exact-Stiefel refinement steps. (3) Compute from the paper's mathematics: the perturbed Gram target, the homotopy schedule, and the predicted original KKT residual scale C delta. Estimate empirically: C from the ratio of measured exact-constraint KKT residual to ||Delta||_F, and transversality from the smallest singular value of the Jacobian of the combined active-support and Gram constraints. (4) First cheap experiment: use a 128x128 sparse orthogonal attention value projection on CIFAR-10 ViT-Tiny or a small WikiText-2 Transformer. Compare exact projection, random perturbation without annealing, and the annealed method at equal sparsity. Log Newton-CG failures, gradient spikes, ||X^T X-I||_F, validation loss, and final KKT residual. The method is supported if it reduces projection/Newton failures during support discovery and, after annealing, reaches equal or better validation accuracy with final orthogonality error below 10^-4.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.