# Progressive Augmented-Lagrangian Warm Starts

- ID: 2200
- Canonical URL: https://synthcore.org/idea/2200/progressive-augmented-lagrangian-warm-starts
- API JSON: https://synthcore.org/api/idea/2200.json
- API Markdown: https://synthcore.org/api/idea/2200.md
- Verification status: unverified
- Source: [arXiv:2608.12665](https://arxiv.org/abs/2608.12665)
- Category: optimization
- Solves: speedup, sample-efficiency, stability
- ML areas: optimizer, training, curriculum, regularization
- Math tags: optimization, dynamical-systems, numerical-analysis
- Ratings: usefulness 6/10; difficulty 5/10; novelty 5/10

## Idea description

Train a constrained neural network on progressively larger data subsets rather than repeatedly solving the full constrained problem from scratch. At each stage, warm-start both the network parameters and constraint multipliers, and use a conservative augmented-Lagrangian gradient update; the paper's local-linear result predicts rapid refinement once the current iterate is near a strong second-order constrained solution.

## Mathematical statement

The paper studies the equality-constrained problem $\min_{x\in\mathbb{R}^{n}} f(x)\ \text{subject to }c(x)=0$, where $x$ is the trainable parameter vector, $f:\mathbb{R}^{n}\to\mathbb{R}$ is the objective, and $c:\mathbb{R}^{n}\to\mathbb{R}^{m}$ is a vector of equality constraints. Use the augmented Lagrangian $\mathcal{L}_{\rho}(x,\lambda)=f(x)+\lambda^{\mathsf T}c(x)+\frac{\rho}{2}\|c(x)\|_{2}^{2}$, with multiplier $\lambda\in\mathbb{R}^{m}$ and penalty $\rho>0$. The paper's local result says that, near a strong second-order stationary point $x^{\star}$, sufficiently small step size $\alpha$ and sufficiently large $\rho$ make the Gradient-Eigenstep method reduce to gradient descent on this augmented Lagrangian and converge locally linearly. The implementable contraction target is $\|x_{k+1}-x^{\star}\|\le q\|x_k-x^{\star}\|$ for some $q<1$ in that local neighborhood; $q$ is not assumed known and should be estimated empirically from successive parameter distances. For a finite-sum problem, define $f_s(x)=|S_s|^{-1}\sum_{i\in S_s}\ell_i(x)$ and $c_s(x)=|S_s|^{-1}\sum_{i\in S_s}c_i(x)$ on a sample subset $S_s$ whose size increases with stage $s$.

## Key formulas

- $$\min_{x\in\mathbb{R}^{n}} f(x)\ \ \text{subject to }c(x)=0$$
- $$\mathcal{L}_{\rho}(x,\lambda)=f(x)+\lambda^{\mathsf T}c(x)+\frac{\rho}{2}\|c(x)\|_{2}^{2},\qquad x_{k+1}=x_k-\alpha\nabla_x\mathcal{L}_{\rho}(x_k,\lambda_k),\qquad \lambda_{k+1}=\lambda_k+\rho c(x_{k+1})$$
- $$f_s(x)=\frac{1}{|S_s|}\sum_{i\in S_s}\ell_i(x),\qquad c_s(x)=\frac{1}{|S_s|}\sum_{i\in S_s}c_i(x),\qquad \|x_{k+1}-x^{\star}\|\le q\|x_k-x^{\star}\|,\quad 0<q<1$$

## Implementation notes

(1) Integrate this into the training loop for a network with differentiable equality constraints, such as layer orthogonality $c(W)=\operatorname{vec}(W^{\mathsf T}W-I)$, fixed feature moments, conservation constraints, or exact output-balance constraints. Maintain network parameters $x$ and a multiplier vector $\lambda$ with the same dimension as the constraint vector. (2) Use progressive sample stages. Start with a small subset $S_0$ and train using the augmented objective $f_s(x)+\lambda^{\mathsf T}c_s(x)+(\rho_s/2)\|c_s(x)\|^2$. For each minibatch or full-stage update, compute $g=\nabla_x f_s(x)+J_{c_s}(x)^{\mathsf T}\lambda+\rho_sJ_{c_s}(x)^{\mathsf T}c_s(x)$, set $x\leftarrow x-\alpha g$, then update $\lambda\leftarrow\lambda+\rho_s c_s(x)$. When the stage terminates, enlarge $S_s$ by a fixed factor such as 2 or 4, retain $(x,\lambda)$, and continue. Increase $\rho_s$ only when constraint violation fails to decrease, for example $\rho_{s+1}=2\rho_s$ if $\|c_s(x)\|>0.9\|c_s(x_{\mathrm{stage-start}})\|$; otherwise keep it fixed. (3) Estimate local contraction empirically using $q_k=\|x_{k+1}-x_k\|/(\|x_k-x_{k-1}\|+10^{-12})$. Reduce $\alpha$ if $q_k>1$ or the augmented objective becomes unstable, and declare a stage converged when $q_k<0.8$ for several updates and the constraint residual is decreasing. The mathematics supplies the augmented-Lagrangian structure and local-linear target; subset sizes, stopping thresholds, and penalty adaptation are engineering choices. (4) First test on CIFAR-10 with a small ResNet-18 and one orthogonality-constrained layer, comparing full-data AdamW plus penalty against progressive augmented-Lagrangian SGD at equal examples, gradient evaluations, and final constraint tolerance. Measure training loss versus examples, constraint violation, wall-clock time, and final accuracy. A positive result is faster loss reduction and lower variance during late-stage refinement, with no accuracy loss at the same final constraint residual; compare against restarting each sample stage from scratch to isolate the warm-start benefit.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.