# Regularity-Gated MGDA

- ID: 959
- Canonical URL: https://synthcore.org/idea/959/regularity-gated-mgda
- API JSON: https://synthcore.org/api/idea/959.json
- API Markdown: https://synthcore.org/api/idea/959.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.15412](https://arxiv.org/abs/2607.15412)
- Category: optimization
- Solves: stability, accuracy, sample-efficiency
- ML areas: optimizer, training, loss
- Math tags: optimization, convex-analysis, linear-algebra, dynamical-systems
- Ratings: usefulness 7/10; difficulty 5/10; novelty 5/10

## Idea description

Replace unconditional stochastic MGDA in a multi-task network with a regularity-gated update. Compute the conflict-avoidant simplex combination when the objective-gradient geometry is sufficiently regular, but use a fixed scalarization weight when the MGDA solution is near a degenerate simplex face or changes sharply between mini-batches. The gate targets the paper's distinction between 1/2-Hölder behavior in the worst case and Lipschitz behavior on regular subproblems.

## Mathematical statement

For model parameters x in R^p and M objective gradients g_m(x) = grad_x f_m(x), define the gradient matrix G = [g_1,...,g_M] in R^(p x M). MGDA chooses simplex coefficients alpha^*(G) = argmin_{alpha in Delta_M} ||G alpha||_2^2, where Delta_M = {alpha in R^M: alpha_m >= 0 and sum_m alpha_m = 1}, and updates with d(G) = -G alpha^*(G). The paper's central regularity statement is that the conflict-avoidant direction can obey ||d(G)-d(H)||_2 <= C ||G-H||_F^(1/2) in the worst case, while under an additional regularity condition it obeys the stronger Lipschitz bound ||d(G)-d(H)||_2 <= L ||G-H||_F. Here G and H are gradient matrices from two nearby iterates or mini-batches, C is a worst-case continuity constant, and L is a local regularity constant. Since the extracted text does not expose the paper's exact regularity predicate, use a computable proxy: the minimum eigenvalue of the Gram matrix K = G^T G restricted to the currently positive MGDA coordinates, the coefficient margin min_{m in A} alpha_m, and the temporal gradient-change ratio. A subproblem is declared regular when lambda_min(K_A) >= epsilon_eig, min_{m in A} alpha_m >= epsilon_alpha, the active set A is unchanged from the previous step, and ||G_t-G_{t-1}||_F/(||G_{t-1}||_F+epsilon) <= epsilon_change. The fallback direction is d_scalar = -G w for a fixed w in Delta_M, normally w_m = 1/M.

## Key formulas

- $$F_S(x)=\bigl(f_{S,1}(x),\dots,f_{S,M}(x)\bigr)^\top,\qquad f_{S,m}(x)=\frac{1}{n}\sum_{i=1}^n f_{z_i,m}(x).$$
- $$\alpha^*(G)=\arg\min_{\alpha\in\Delta_M}\|G\alpha\|_2^2,\qquad \Delta_M=\{\alpha\in\mathbb{R}^M:\alpha_m\ge 0,\ \mathbf{1}^\top\alpha=1\},\qquad d_{\mathrm{CA}}(G)=-G\alpha^*(G).$$
- $$\|d_{\mathrm{CA}}(G)-d_{\mathrm{CA}}(H)\|_2\le C\|G-H\|_F^{1/2}\quad\text{in general},\qquad \|d_{\mathrm{CA}}(G)-d_{\mathrm{CA}}(H)\|_2\le L\|G-H\|_F\quad\text{on regular subproblems}.$$$
- $$d_t=\begin{cases}-G_t\alpha_t^*,&\lambda_{\min}\bigl((G_t^\top G_t)_{A_t,A_t}\bigr)\ge\varepsilon_{\mathrm{eig}},\ \min_{m\in A_t}\alpha_{t,m}^*\ge\varepsilon_\alpha,\ A_t=A_{t-1},\ r_t\le\varepsilon_{\mathrm{change}},\\-G_t w,&\text{otherwise},\end{cases}\qquad r_t=\frac{\|G_t-G_{t-1}\|_F}{\|G_{t-1}\|_F+\epsilon},\qquad w_m=1/M.$$$

## Implementation notes

Integrate this into the shared-parameter optimizer of a multi-task network, immediately after computing one gradient per task and before the optimizer update. For a batch B_t, compute g_{t,m}=grad_x f_m(B_t;x_t), optionally flattening only the shared trunk parameters; stack them as G_t=[g_{t,1},...,g_{t,M}]. Solve the small quadratic program alpha_t=argmin_{alpha>=0, 1^T alpha=1} alpha^T(G_t^T G_t)alpha using an active-set QP solver or projected-gradient iterations. Let A_t={m:alpha_{t,m}>epsilon_alpha}. Maintain G_{t-1}, alpha_{t-1}, and A_{t-1}; for numerical robustness, calculate the eigenvalue of the active Gram submatrix with torch.linalg.eigvalsh, and use an exponential moving average Gbar_t=beta Gbar_{t-1}+(1-beta)G_t for the gate only, while applying the update direction from the current gradients. Set r_t=||Gbar_t-Gbar_{t-1}||_F/(||Gbar_{t-1}||_F+1e-8). If the active-set, eigenvalue, coefficient-margin, and r_t tests all pass, apply d_t=-G_t alpha_t. Otherwise apply d_t=-G_t w with uniform w, or a fixed task-priority vector if the application has a known scalarization. Feed d_t into AdamW or SGD as the shared-trunk gradient; task-specific heads can retain their ordinary task gradients. Stop-gradient through alpha and through the gate. The paper supplies the continuity rationale, while epsilon_eig, epsilon_alpha, epsilon_change, beta, and the scalarization vector are empirical hyperparameters. First test on a two- or five-task NYUv2, MultiMNIST, or synthetic conflicting-quadratics benchmark using a small MLP or ResNet, comparing uniform scalarization, vanilla stochastic MGDA, and PCGrad at equal optimizer steps and backward passes. Track mean validation objective, worst-task validation loss, cosine conflict rate, ||d_t-d_{t-1}||, and the fraction of regular versus fallback steps. Success is a smoother direction norm and faster decrease of the worst-task loss, with improved performance over vanilla MGDA at equal FLOPs; a useful diagnostic is convergence closer to a T^{-1/2}-style slope rather than the noisy T^{-1/4}-style slope reported for stochastic MGDA.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built an exact small-M MGDA solver, EMA-based Regularity-Gated MGDA, numerical math checks, and a reproducible noisy conflicting-quadratics benchmark. The regular interior sweep showed the predicted Lipschitz scaling (log-log slope 1.040 vs 1.0), and gate activation decreased monotonically as noise increased (0.99, 0.99, 0.83, 0.33 for noise 0.01, 0.05, 0.15, 0.4). However, the degenerate zero-curvature construction showed a slope 1.025 rather than the predicted 0.5, and the gated optimizer did not improve over uniform scalarization (mean final loss 1.7780 vs 1.7780; worst-task loss 2.24085 vs 2.24085), so the full mechanism is not validated.

### Mechanism check

- Verdict: Built an exact small-M MGDA solver, EMA-based Regularity-Gated MGDA, numerical math checks, and a reproducible noisy conflicting-quadratics benchmark. The regular interior sweep showed the predicted Lipschitz scaling (log-log slope 1.040 vs 1.0), and gate activation decreased monotonically as noise increased (0.99, 0.99, 0.83, 0.33 for noise 0.01, 0.05, 0.15, 0.4). However, the degenerate zero-curvature construction showed a slope 1.025 rather than the predicted 0.5, and the gated optimizer did not improve over uniform scalarization (mean final loss 1.7780 vs 1.7780; worst-task loss 2.24085 vs 2.24085), so the full mechanism is not validated.
- Confidence: 9/10
- Limitations: This is a small synthetic quadratic benchmark, not a real multi-task neural network or dataset; PCGrad was not included. The proxy gate uses active-Gram curvature rather than the paper’s exact reduced tangent-space predicate, and the toy degenerate sweep did not reproduce the worst-case 1/2-Hölder family.

## Artifacts

- [experiment.py](https://synthcore.org/code/403/experiment.py)
- [report.md](https://synthcore.org/code/403/report.md)
- [results.json](https://synthcore.org/code/403/results.json)
- [results_final.json](https://synthcore.org/code/403/results_final.json)
- [Download all files as ZIP](https://synthcore.org/download/403)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.