Residual-Adaptive Manifold-Affine Damping
Implementation & benchmark of arXiv:2606.31738 — A Geometry-Adaptive Regularized Newton-Type Method for Manifold-Affine Intersection Problems
Source paper: A Geometry-Adaptive Regularized Newton-Type Method for Manifold-Affine Intersection Problems arXiv:2606.31738 ⓘ · analyzed Aug 29, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace fixed-strength projection or constraint-repair steps during low-rank neural fine-tuning with a regularized affine subproblem whose damping is proportional to the current distance from the model manifold. Use strong damping when a gradient update leaves the low-rank manifold substantially, then automatically remove the damping near a clean intersection so that the method can recover higher-order local convergence. This is suitable for LoRA-style updates, structured matrix compression, and fine-tuning subject to linear calibration or output-preservation constraints.
Formulas
Mathematical statement
The paper defines the manifold residual as $r_k=d_M(x_k)=\|x_k-P_M(x_k)\|$, where $M$ is the constraint manifold, $P_M$ is a nearest-point manifold projection, and $x_k$ is the current iterate. Its central rule is $\mu_k=c r_k^{\rho}$ with $c>0$ and $\rho\in[0,1]$: fixed regularization when $\rho=0$, and asymptotically vanishing regularization when $\rho>0$. Under intrinsic transversality the method is locally linearly convergent; under ordinary transversality its local order is $1+\rho$, hence quadratic for $\rho=1$. For a rank-constrained neural matrix, let $W\in\mathbb R^{m\times n}$ lie on the rank-$q$ manifold $M_q$, let the affine constraints be $B\operatorname{vec}(Z)=b$, and let $Y_k=P_{M_q}(X_k)$ be the rank-$q$ truncated-SVD projection of the post-gradient matrix $X_k$. If $Y_k=U\Sigma V^\top$, the tangent projector is $P_T(H)=UU^\top H+HVV^\top-UU^\top HVV^\top$ and the normal projector is $P_N(H)=H-P_T(H)$. Use the adapted regularized intersection step $Z_k=\arg\min_{B\operatorname{vec}(Z)=b}\frac12\|P_N(Z-Y_k)\|_F^2+\frac{\mu_k}{2}\|Z-X_k\|_F^2$. The normal penalty captures local manifold geometry, while the quadratic term makes the affine solve nonsingular when the restricted tangent operator is rank deficient. Define $r_k=\|X_k-Y_k\|_F/(\|X_k\|_F+10^{-12})$ and use $\mu_k=c(r_k+\epsilon)^\rho$ with small numerical $\epsilon>0$.
Implementation notes
Integrate this after the optimizer update of a matrix-valued layer, rather than inside the forward pass. Start with one LoRA or low-rank linear layer. Given weights $W_k$, compute the ordinary AdamW or SGD update $X_k=W_k-\eta g_k$. Compute the rank-$q$ truncated SVD $Y_k=U\Sigma V^\top$ of $X_k$, and set $r_k=\|X_k-Y_k\|_F/(\|X_k\|_F+10^{-12})$. Choose $\mu_k=c(r_k+10^{-6})^\rho$, initially testing $(c,\rho)\in\{(1,0),(1,0.5),(1,1),(10,1)\}$. Implement matrix-free applications of $P_N(H)=H-(UU^\top H+HVV^\top-UU^\top HVV^\top)$. Solve the equality-constrained quadratic problem with a KKT system or projected conjugate gradients. The KKT equations are $\begin{bmatrix}Q_k&B^\top\\B&0\end{bmatrix}\begin{bmatrix}\operatorname{vec}(Z_k)\\\lambda\end{bmatrix}=\begin{bmatrix}\mu_k\operatorname{vec}(X_k)+Q_k\operatorname{vec}(Y_k)\\b\end{bmatrix}$, where $Q_k=P_N^\ast P_N+\mu_k I$ and $\lambda$ enforces the affine constraint. Set $W_{k+1}=P_{M_q}(Z_k)$, or retain $Z_k$ when affine feasibility is prioritized and periodically reproject to $M_q$. The paper supplies the residual rule and tangent-normal quadratic; estimate SVD quantities and solve tolerances numerically. The first cheap experiment is rank-4 fine-tuning of a $768\times768$ layer on CIFAR-10 or GLUE, with synthetic linear constraints preserving outputs on 32 calibration examples. Compare AdamW, fixed-$\mu$ projection, alternating SVD/affine projections, and adaptive damping at equal optimizer steps and SVD calls. Record task loss, rank residual, affine residual, KKT failures, and wall-clock time. Success means fewer projection oscillations, monotone residual reduction near degenerate tangent intersections, and better validation accuracy at equal projection cost.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built a readable MVP implementing truncated-SVD rank projection, tangent/normal projectors, the affine KKT subproblem, residual-adaptive damping, and comparisons against fixed damping and alternating projections. The projector identities and predicted mu-versus-residual power law passed numerically, but the optimization benchmark showed no promised practical win: fixed damping and alternating projections reached approximately 1e-15 target error, while adaptive rho=1 ended at approximately 2.6e-10 and was much slower due to explicit KKT construction.
- Agent confidence
- 9/10
- Baseline
- Alternating projections: final target error 9.09e-16, affine residual 1.60e-15, rank residual 2.91e-16, 0.10 s; fixed mu=1: target error 9.86e-16, affine residual 1.83e-15, rank residual 6.22e-16, 10.18 s.
- Idea
- Adaptive rho=0.5: final target error 3.46e-13; adaptive rho=1: 2.56e-10; adaptive c=10,rho=1: 3.63e-11. All affine and rank residuals were about 1e-15 or lower, but none beat the controls on target error.
Limitations: This was a small synthetic 8x7 rank-2 matrix problem with 12 random linear constraints and 100 identical gradient steps, not a neural fine-tuning or CIFAR/GLUE experiment. The KKT matrix was formed explicitly, so runtime is not representative of a matrix-free production implementation; no GPU, AdamW, noisy gradients, degenerate intersections, validation accuracy, or higher-order convergence-rate fit was tested.
How to run: python3 experiment.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Artifacts
- experiment.py 4.8 KB View Raw
- report.md 1.7 KB View
- results.json 89.4 KB View Raw