Regularity-Gated MGDA

Integrate this into the shared-parameter optimizer of a multi-task network, immediately after computing one gradient per task and before the optimizer update. For a batch B_t, compute g_{t,m}=grad_x f_m(B_t;x_t), optionally flattening only the shared trunk parameters; stack them as G_t=[g_{t,1},...,g_{t,M}]. Solve the small quadratic program alpha_t=argmin_{alpha>=0, 1^T alpha=1} alpha^T(G_t^T G_t)alpha using an active-set QP solver or projected-gradient iterations. Let A_t={m:alpha_{t,m}>epsilon_alpha}. Maintain G_{t-1}, alpha_{t-1}, and A_{t-1}; for numerical robustness, calculate the eigenvalue of the active Gram submatrix with torch.linalg.eigvalsh, and use an exponential moving average Gbar_t=beta Gbar_{t-1}+(1-beta)G_t for the gate only, while applying the update direction from the current gradients. Set r_t=||Gbar_t-Gbar_{t-1}||_F/(||Gbar_{t-1}||_F+1e-8). If the active-set, eigenvalue, coefficient-margin, and r_t tests all pass, apply d_t=-G_t alpha_t. Otherwise apply d_t=-G_t w with uniform w, or a fixed task-priority vector if the application has a known scalarization. Feed d_t into AdamW or SGD as the shared-trunk gradient; task-specific heads can retain their ordinary task gradients. Stop-gradient through alpha and through the gate. The paper supplies the continuity rationale, while epsilon_eig, epsilon_alpha, epsilon_change, beta, and the scalarization vector are empirical hyperparameters. First test on a two- or five-task NYUv2, MultiMNIST, or synthetic conflicting-quadratics benchmark using a small MLP or ResNet, comparing uniform scalarization, vanilla stochastic MGDA, and PCGrad at equal optimizer steps and backward passes. Track mean validation objective, worst-task validation loss, cosine conflict rate, ||d_t-d_{t-1}||, and the fraction of regular versus fallback steps. Success is a smoother direction norm and faster decrease of the worst-task loss, with improved performance over vanilla MGDA at equal FLOPs; a useful diagnostic is convergence closer to a T^{-1/2}-style slope rather than the noisy T^{-1/4}-style slope reported for stochastic MGDA.