# Spectral-filtered task-gradient optimizer

- ID: 370
- Canonical URL: https://synthcore.org/idea/370/spectral-filtered-task-gradient-optimizer
- API JSON: https://synthcore.org/api/idea/370.json
- API Markdown: https://synthcore.org/api/idea/370.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.02681](https://arxiv.org/abs/2607.02681)
- Category: optimization
- Solves: stability, accuracy, generalization
- ML areas: optimizer, federated
- Math tags: robust-statistics, probability, linear-algebra, optimization
- Ratings: usefulness 8/10; difficulty 5/10; novelty 5/10

## Idea description

Replace the ordinary average of task or client gradients with an iterative spectral filter that removes tasks whose gradient vectors explain an anomalously large covariance direction. The global model uses the filtered gradient, while each task still maintains its own personalized parameters and local optimizer state. Unlike parameter-center regularization, the robustification acts directly on the vector messages and is designed to avoid an additional \(\sqrt d\) contamination penalty.

## Mathematical statement

The paper's mean-estimation example uses squared loss \(\ell(\theta,z)=\|\theta-z\|_2^2/2\), for which \(\nabla\ell(\theta,z)=\theta-z\) and the population risk is strongly convex and smooth with constants \(L=L'=1\). Its proposed algorithms operate through robust aggregation of task-level gradients. The implementable adaptation is a spectral robust mean procedure: for task gradients \(g_k\in\mathbb R^d\), maintain an active set \(S\subseteq\{1,\ldots,K\}\), compute the active mean \(\mu_S=|S|^{-1}\sum_{k\in S}g_k\) and covariance \(C_S=|S|^{-1}\sum_{k\in S}(g_k-\mu_S)(g_k-\mu_S)^\top\). Let \((\hat v,\hat\lambda)\) be the top unit eigenvector/eigenvalue of \(C_S\). If \(\hat\lambda\) exceeds the trusted inlier variance threshold \(\tau^2[1+c\sqrt{\log(1/\delta)/|S|}]\), remove the task with largest score \(r_k=|\hat v^\top(g_k-\mu_S)|\), or remove all tasks with \(r_k>\tau\sqrt{2\log(|S|/\delta)}\); otherwise return \(\mu_S\). Here \(\tau\) is an estimated inlier gradient scale, \(c\) is a numerical constant, \(\delta\) is the failure probability, \(K\) is the number of tasks, and \(d\) is parameter dimension. The spectral test targets covariance inflation caused by contaminated vectors rather than using coordinatewise clipping, whose error can accumulate as \(\sqrt d\).

## Key formulas

- $$\ell(\theta,z)=\frac{1}{2}\|\theta-z\|_2^2,\qquad \nabla\ell(\theta,z)=\theta-z,\qquad \nabla^2\ell(\theta,z)=I_d.$$
- $$\mu_S=\frac{1}{|S|}\sum_{k\in S}g_k,\qquad C_S=\frac{1}{|S|}\sum_{k\in S}(g_k-\mu_S)(g_k-\mu_S)^\top.$$
- $$\hat\lambda=\lambda_{\max}(C_S)\leq \tau^2\left(1+c\sqrt{\frac{\log(1/\delta)}{|S|}}\right)\ \Longrightarrow\ \widehat g=\mu_S;\qquad \hat\lambda>\tau^2\left(1+c\sqrt{\frac{\log(1/\delta)}{|S|}}\right)\ \Longrightarrow\ S\leftarrow S\setminus\left\{\arg\max_{k\in S}\left|\hat v^\top(g_k-\mu_S)\right|\right\}.$$
- $$\theta_{t+1}=\theta_t-\eta_t\widehat g_t,\qquad \widehat g_t=\operatorname{SpectralFilter}(g_{1,t},\ldots,g_{K,t}),\qquad g_{k,t}=\frac{1}{b}\sum_{i\in B_{k,t}}\nabla_\theta\ell(\theta_t,z_i^{(k)}).$$

## Implementation notes

Integrate this into synchronous federated or multi-task training immediately before the global optimizer update. At iteration \(t\), broadcast the current global parameters \(\theta_t\); each task or client forms a minibatch gradient \(g_{k,t}\in\mathbb R^d\), and up to an \(\epsilon\) fraction may be arbitrary. Do not concatenate all examples. Instead run the following filter on the \(K\) vectors: initialize \(S=\{1,\ldots,K\}\); estimate the inlier scale \(\tau\) from a trusted warm-up period using the median of per-task gradient norms or from the median coordinatewise variance; compute \(\mu_S\), the covariance-vector product operator \(u\mapsto C_Su\), and its top eigenpair by 3--5 power iterations; if the eigenvalue test in the formula fails, remove the task with largest projection score and repeat; stop when the covariance passes the threshold or when \(|S|<(1-2\epsilon)K\). Use the surviving mean \(\widehat g_t\) in AdamW or SGD, while keeping ordinary local or personalized heads updated from each task's own gradient. For memory efficiency, never materialize \(C_S\): compute \(C_Su=|S|^{-1}\sum_{k\in S}(g_k-\mu_S)((g_k-\mu_S)^\top u)\). The mathematical quantities are the mean, covariance, leading eigenvector, and rejection threshold; \(\tau\), the contamination fraction, and the threshold constant should be estimated or tuned empirically. First test on a 2-layer MLP trained across 20 synthetic Gaussian-mean tasks in dimension 128, with 10--30% clients replaced by sign-flipped or randomly shifted gradients. Compare SGD averaging, coordinatewise clipping, median aggregation, and this filter at equal communication and FLOPs. Measure clean-task excess risk, global loss, personalization error, and the error of the aggregate gradient relative to the uncontaminated mean. Success means substantially smaller gradient bias and stable convergence as contamination increases, with no degradation in the uncontaminated regime; a useful target is retaining near-clean accuracy at 20% corruption where ordinary averaging diverges.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built an iterative matrix-free spectral gradient filter and compared it with mean, coordinate median, and coordinate clipping on 20-task d=128 quadratic federated simulations. The covariance matvec identity was verified to 1.1e-16 error and a synthetic outlier was removed, but the benchmark showed no win: at 0–20% corruption the filter had higher excess loss (0.00225, 0.00348, 0.00371) than mean aggregation (0.00017, 0.00032, 0.00047) and removed about 9 of 20 clients even without corruption. A targeted sweep showed that increasing tau avoids false rejection but then fails to reject aligned attacks, so this MVP does not establish the promised effect.

### Mechanism check

- Verdict: Built an iterative matrix-free spectral gradient filter and compared it with mean, coordinate median, and coordinate clipping on 20-task d=128 quadratic federated simulations. The covariance matvec identity was verified to 1.1e-16 error and a synthetic outlier was removed, but the benchmark showed no win: at 0–20% corruption the filter had higher excess loss (0.00225, 0.00348, 0.00371) than mean aggregation (0.00017, 0.00032, 0.00047) and removed about 9 of 20 clients even without corruption. A targeted sweep showed that increasing tau avoids false rejection but then fails to reject aligned attacks, so this MVP does not establish the promised effect.
- Confidence: 9/10
- Limitations: This is a synthetic quadratic mean-estimation benchmark rather than the planned two-layer MLP with personalized heads. The inlier scale was fixed rather than learned from a trusted warm-up, and baselines were not matched with a full communication/FLOP accounting or extensive random-seed confidence intervals.

## Artifacts

- [report.md](https://synthcore.org/code/55/report.md)
- [results.json](https://synthcore.org/code/55/results.json)
- [spectral_filter_experiment.py](https://synthcore.org/code/55/spectral_filter_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/55)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.