# Residual-Adaptive Manifold-Affine Damping

- ID: 284
- Canonical URL: https://synthcore.org/idea/284/residual-adaptive-manifold-affine-damping
- API JSON: https://synthcore.org/api/idea/284.json
- API Markdown: https://synthcore.org/api/idea/284.md
- Verification status: mechanism_failed
- Source: [arXiv:2606.31738](https://arxiv.org/abs/2606.31738)
- Category: optimization
- Solves: stability, accuracy, sample-efficiency
- ML areas: optimizer, fine-tuning, regularization, mlp
- Math tags: optimization, geometry, differential-geometry, linear-algebra, dynamical-systems
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Replace fixed-strength projection or constraint-repair steps during low-rank neural fine-tuning with a regularized affine subproblem whose damping is proportional to the current distance from the model manifold. Use strong damping when a gradient update leaves the low-rank manifold substantially, then automatically remove the damping near a clean intersection so that the method can recover higher-order local convergence. This is suitable for LoRA-style updates, structured matrix compression, and fine-tuning subject to linear calibration or output-preservation constraints.

## Mathematical statement

The paper defines the manifold residual as $r_k=d_M(x_k)=\|x_k-P_M(x_k)\|$, where $M$ is the constraint manifold, $P_M$ is a nearest-point manifold projection, and $x_k$ is the current iterate. Its central rule is $\mu_k=c r_k^{\rho}$ with $c>0$ and $\rho\in[0,1]$: fixed regularization when $\rho=0$, and asymptotically vanishing regularization when $\rho>0$. Under intrinsic transversality the method is locally linearly convergent; under ordinary transversality its local order is $1+\rho$, hence quadratic for $\rho=1$. For a rank-constrained neural matrix, let $W\in\mathbb R^{m\times n}$ lie on the rank-$q$ manifold $M_q$, let the affine constraints be $B\operatorname{vec}(Z)=b$, and let $Y_k=P_{M_q}(X_k)$ be the rank-$q$ truncated-SVD projection of the post-gradient matrix $X_k$. If $Y_k=U\Sigma V^\top$, the tangent projector is $P_T(H)=UU^\top H+HVV^\top-UU^\top HVV^\top$ and the normal projector is $P_N(H)=H-P_T(H)$. Use the adapted regularized intersection step $Z_k=\arg\min_{B\operatorname{vec}(Z)=b}\frac12\|P_N(Z-Y_k)\|_F^2+\frac{\mu_k}{2}\|Z-X_k\|_F^2$. The normal penalty captures local manifold geometry, while the quadratic term makes the affine solve nonsingular when the restricted tangent operator is rank deficient. Define $r_k=\|X_k-Y_k\|_F/(\|X_k\|_F+10^{-12})$ and use $\mu_k=c(r_k+\epsilon)^\rho$ with small numerical $\epsilon>0$.

## Key formulas

- $$\mu_k=c\,r_k^{\rho},\qquad r_k:=d_M(x_k)=\|x_k-P_M(x_k)\|,\qquad c>0,\quad \rho\in[0,1].$$
- $$\operatorname{ord}(x_{k+1}-x^*)=1+\rho,\qquad\text{in particular }\rho=1\Rightarrow\text{quadratic local convergence under transversality}.$$
- $$P_T(H)=UU^\top H+HVV^\top-UU^\top HVV^\top,\qquad P_N(H)=H-P_T(H),$$
- $$Z_k=\arg\min_{B\operatorname{vec}(Z)=b}\frac12\left\|P_N(Z-Y_k)\right\|_F^2+\frac{\mu_k}{2}\left\|Z-X_k\right\|_F^2,\qquad Y_k=P_{M_q}(X_k).$$

## Implementation notes

Integrate this after the optimizer update of a matrix-valued layer, rather than inside the forward pass. Start with one LoRA or low-rank linear layer. Given weights $W_k$, compute the ordinary AdamW or SGD update $X_k=W_k-\eta g_k$. Compute the rank-$q$ truncated SVD $Y_k=U\Sigma V^\top$ of $X_k$, and set $r_k=\|X_k-Y_k\|_F/(\|X_k\|_F+10^{-12})$. Choose $\mu_k=c(r_k+10^{-6})^\rho$, initially testing $(c,\rho)\in\{(1,0),(1,0.5),(1,1),(10,1)\}$. Implement matrix-free applications of $P_N(H)=H-(UU^\top H+HVV^\top-UU^\top HVV^\top)$. Solve the equality-constrained quadratic problem with a KKT system or projected conjugate gradients. The KKT equations are $\begin{bmatrix}Q_k&B^\top\\B&0\end{bmatrix}\begin{bmatrix}\operatorname{vec}(Z_k)\\\lambda\end{bmatrix}=\begin{bmatrix}\mu_k\operatorname{vec}(X_k)+Q_k\operatorname{vec}(Y_k)\\b\end{bmatrix}$, where $Q_k=P_N^\ast P_N+\mu_k I$ and $\lambda$ enforces the affine constraint. Set $W_{k+1}=P_{M_q}(Z_k)$, or retain $Z_k$ when affine feasibility is prioritized and periodically reproject to $M_q$. The paper supplies the residual rule and tangent-normal quadratic; estimate SVD quantities and solve tolerances numerically. The first cheap experiment is rank-4 fine-tuning of a $768\times768$ layer on CIFAR-10 or GLUE, with synthetic linear constraints preserving outputs on 32 calibration examples. Compare AdamW, fixed-$\mu$ projection, alternating SVD/affine projections, and adaptive damping at equal optimizer steps and SVD calls. Record task loss, rank residual, affine residual, KKT failures, and wall-clock time. Success means fewer projection oscillations, monotone residual reduction near degenerate tangent intersections, and better validation accuracy at equal projection cost.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a readable MVP implementing truncated-SVD rank projection, tangent/normal projectors, the affine KKT subproblem, residual-adaptive damping, and comparisons against fixed damping and alternating projections. The projector identities and predicted mu-versus-residual power law passed numerically, but the optimization benchmark showed no promised practical win: fixed damping and alternating projections reached approximately 1e-15 target error, while adaptive rho=1 ended at approximately 2.6e-10 and was much slower due to explicit KKT construction.

### Mechanism check

- Verdict: Built a readable MVP implementing truncated-SVD rank projection, tangent/normal projectors, the affine KKT subproblem, residual-adaptive damping, and comparisons against fixed damping and alternating projections. The projector identities and predicted mu-versus-residual power law passed numerically, but the optimization benchmark showed no promised practical win: fixed damping and alternating projections reached approximately 1e-15 target error, while adaptive rho=1 ended at approximately 2.6e-10 and was much slower due to explicit KKT construction.
- Confidence: 9/10
- Limitations: This was a small synthetic 8x7 rank-2 matrix problem with 12 random linear constraints and 100 identical gradient steps, not a neural fine-tuning or CIFAR/GLUE experiment. The KKT matrix was formed explicitly, so runtime is not representative of a matrix-free production implementation; no GPU, AdamW, noisy gradients, degenerate intersections, validation accuracy, or higher-order convergence-rate fit was tested.

## Artifacts

- [experiment.py](https://synthcore.org/code/1254/experiment.py)
- [report.md](https://synthcore.org/code/1254/report.md)
- [results.json](https://synthcore.org/code/1254/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1254)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.