Proximal Regularized Extragradient for Sparse Adapters

Insert the method into LoRA or a small trainable adapter while freezing the backbone. Let $x$ contain only adapter weights, $F(x)=\nabla L_{\mathrm{train}}(x)$, $H(x)=L_{\mathrm{val}}(x)$, and $R(x)=\rho\sum_j\|x_{G_j}\|_2$ for predefined LoRA rank groups $G_j$. Its exact proximal map is group soft-thresholding: $\operatorname{prox}_{\eta R}(z)_{G_j}=\max(0,1-\eta\rho/\|z_{G_j}\|_2)z_{G_j}$. For each step, calculate training and validation gradients at $x$, set $q_x=g_x+\lambda_kv_x$, and form $y=\operatorname{prox}_{\eta R}(x-\eta q_x)$. Recompute gradients at $y$, set $q_y=g_y+\lambda_kv_y$, and update $x\leftarrow\operatorname{prox}_{\eta R}(x-\eta q_y)$. Use separate train and validation minibatches, $\lambda_k=\lambda_0(k+1)^{-p}$, and apply the proximal map only to adapter groups if biases should remain dense. The mathematical assumptions are convexity of $R$ and approximate monotonicity of $F$; estimate monotonicity from random parameter pairs, while the proximal map and regularization schedule are exact. Test on a frozen-encoder AG News or CIFAR-10 classifier with rank-8 LoRA, comparing AdamW plus group-lasso, proximal SGD, and ordinary extragradient at matched forward/backward counts. Measure validation accuracy, active adapter groups, adapter memory, and iterate oscillation. Success is equal or better validation accuracy with at least 25% fewer active groups or lower adapter memory and no worse last-iterate loss.