# Proximal Regularized Extragradient for Sparse Adapters

- ID: 2815
- Canonical URL: https://synthcore.org/idea/2815/proximal-regularized-extragradient-for-sparse-adapters
- API JSON: https://synthcore.org/api/idea/2815.json
- API Markdown: https://synthcore.org/api/idea/2815.md
- Verification status: unverified
- Source: [arXiv:2608.29181](https://arxiv.org/abs/2608.29181)
- Category: optimization
- Solves: stability, memory, generalization
- ML areas: optimizer, fine-tuning, pruning, memory
- Math tags: optimization, convex-analysis, dynamical-systems
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Extend regularized extragradient with proximal operators so nonsmooth penalties such as group sparsity, nuclear norms, or parameter constraints are applied at both prediction and correction stages. This can produce sparse or low-rank adapters while retaining the look-ahead stabilization for the smooth inner residual.

## Mathematical statement

The paper's nonsmooth formulation allows an outer objective $H(x)=h(x)+\hat h(x)$ and an inner inclusion $0\in V(x)+\partial\hat f(x)$, where $\partial$ is the convex subdifferential. The adaptation uses the proximal map $\operatorname{prox}_{\eta R}(z)=\arg\min_u\{R(u)+(2\eta)^{-1}\|u-z\|^2\}$ for a proper convex regularizer $R$. Let $F(x)$ be the smooth inner residual and define $G_k(x)=F(x)+\lambda_k\nabla H(x)$. The proximal-extragradient prediction and correction are $y_k=\operatorname{prox}_{\eta R}(x_k-\eta G_k(x_k))$ and $x_{k+1}=\operatorname{prox}_{\eta R}(x_k-\eta G_k(y_k))$. Here $R$ can be group-$\ell_1$, a nuclear norm on a low-rank adapter, or the indicator of an allowed parameter set. The proximal map is nonexpansive for proper convex lower-semicontinuous $R$, and $\lambda_k=\lambda_0(k+1)^{-p}$ supplies decaying outer selection.

## Key formulas

- $$H(x):=h(x)+\hat{h}(x),\qquad 0\in V(x)+\partial\hat{f}(x).$$
- $$\operatorname{prox}_{\eta R}(z)=\arg\min_u\left\{R(u)+\frac{1}{2\eta}\|u-z\|^2\right\}.$$
- $$G_k(x)=F(x)+\lambda_k\nabla H(x),\qquad \lambda_k=\lambda_0(k+1)^{-p}.$$
- $$y_k=\operatorname{prox}_{\eta R}\bigl(x_k-\eta G_k(x_k)\bigr),\qquad x_{k+1}=\operatorname{prox}_{\eta R}\bigl(x_k-\eta G_k(y_k)\bigr).$$

## Implementation notes

Insert the method into LoRA or a small trainable adapter while freezing the backbone. Let $x$ contain only adapter weights, $F(x)=\nabla L_{\mathrm{train}}(x)$, $H(x)=L_{\mathrm{val}}(x)$, and $R(x)=\rho\sum_j\|x_{G_j}\|_2$ for predefined LoRA rank groups $G_j$. Its exact proximal map is group soft-thresholding: $\operatorname{prox}_{\eta R}(z)_{G_j}=\max(0,1-\eta\rho/\|z_{G_j}\|_2)z_{G_j}$. For each step, calculate training and validation gradients at $x$, set $q_x=g_x+\lambda_kv_x$, and form $y=\operatorname{prox}_{\eta R}(x-\eta q_x)$. Recompute gradients at $y$, set $q_y=g_y+\lambda_kv_y$, and update $x\leftarrow\operatorname{prox}_{\eta R}(x-\eta q_y)$. Use separate train and validation minibatches, $\lambda_k=\lambda_0(k+1)^{-p}$, and apply the proximal map only to adapter groups if biases should remain dense. The mathematical assumptions are convexity of $R$ and approximate monotonicity of $F$; estimate monotonicity from random parameter pairs, while the proximal map and regularization schedule are exact. Test on a frozen-encoder AG News or CIFAR-10 classifier with rank-8 LoRA, comparing AdamW plus group-lasso, proximal SGD, and ordinary extragradient at matched forward/backward counts. Measure validation accuracy, active adapter groups, adapter memory, and iterate oscillation. Success is equal or better validation accuracy with at least 25% fewer active groups or lower adapter memory and no worse last-iterate loss.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.