Parameter-Free Certified Augmented-Lagrangian Fine-Tuning

(1) Integration point: implement this as an optimizer for a parameter tensor $z$ in a constrained fine-tuning problem. Start with a linear probe or LoRA adapter so that the task loss is close to convex, and impose exact linear constraints $Az=b$ such as tying two adapter blocks, forcing selected filter sums to zero, or enforcing a fixed affine subspace. Use $\psi_s(z)$ for the minibatch loss and $\psi_n(z)$ for a box or group-lasso proximal term; for a full neural network, treat the method as a heuristic because the paper assumes convexity.

(2) Pseudocode: initialize $z_0$, $y_0=0$, $\rho=\rho_0$, target $\epsilon_0$, and acceleration state $q_0=z_0$. For each outer iteration, define $g(z)=\nabla\psi_s(z)+A^T(y+\rho(Az-b))$. Run accelerated proximal-gradient steps with backtracking: choose $\eta$ by testing sufficient decrease, compute $v=\operatorname{prox}_{\eta\psi_n}(q-\eta g(q))$, update the Nesterov momentum, and estimate $G=(q-v)/\eta$. Stop the inner loop when $\|G\|\le\epsilon_t$ or when a compute cap is reached. Set $z_{t+1}=v$, update $y_{t+1}=y_t+\rho(Az_{t+1}-b)$, and measure $r_{t+1}=\|Az_{t+1}-b\|$. If $r_{t+1}>0.9r_t$, set $\rho\leftarrow2\rho$; otherwise retain $\rho$. Reduce the inner target only when feasibility improves: set $\epsilon_{t+1}=\max(\epsilon_{\min},0.5\epsilon_t)$ if $r_{t+1}\le0.5r_t$, otherwise leave it unchanged. Warm-start every inner solve from the previous $z_{t+1}$.

(3) Computed from the paper's mathematics: the augmented-Lagrangian gradient, multiplier update, primal residual, proximal-gradient stationarity certificate, and adaptive penalty/accuracy logic. Estimated empirically: the local smoothness constant through backtracking, the constraint spectrum through occasional power iteration if preconditioning is desired, and unknown quantities such as $D$, $\bar d$, $M_n$, and $\nabla_s$; do not use them to set hyperparameters.

(4) First cheap experiment: train a 10-class MNIST or CIFAR-10 linear classifier and a rank-8 LoRA adapter on a frozen small MLP, comparing AdamW plus a quadratic constraint penalty, standard fixed-$\rho$ AL, and this certified parameter-free AL at equal gradient evaluations. Use constraints such as $Az=0$ for tied or zero-sum adapter parameters. Record task loss, $\|Az-b\|$, $\|G\|$, wall-clock time, and the number of backtracking evaluations. The expected signal is lower constraint violation at equal task accuracy, fewer manual penalty restarts, and faster decrease of the joint KKT residual; a useful success threshold is at least 2x fewer gradient evaluations than fixed-penalty AL to reach both $\|Az-b\|\le10^{-3}$ and $\|G\|\le10^{-3}$.