# Quadratic Client Legacies

- ID: 659
- Canonical URL: https://synthcore.org/idea/659/quadratic-client-legacies
- API JSON: https://synthcore.org/api/idea/659.json
- API Markdown: https://synthcore.org/api/idea/659.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.09939](https://arxiv.org/abs/2607.09939)
- Category: training
- Solves: stability, sample-efficiency, scalability
- ML areas: optimizer, training, fine-tuning
- Math tags: optimization, convex-analysis, dynamical-systems
- Ratings: usefulness 7/10; difficulty 4/10; novelty 7/10

## Idea description

When a federated or decentralized client leaves, transmit a small gradient-anchored quadratic surrogate instead of discarding its loss. The surrogate preserves the client's gradient exactly at the departure model and supplies a controlled approximation away from that point, allowing training to retain information from unavailable clients with constant memory and communication.

## Mathematical statement

The paper's optimization problem is $F(x)=\sum_{i=1}^{n}f_i(x)$, where $x\in\mathbb{R}^d$ is the shared model and $f_i$ is client $i$'s private loss. For a departing client $i$, choose an anchor $a_i$ equal to the shared parameters at departure, gradient $g_i=\nabla f_i(a_i)$, and quadratic curvature $\kappa_i>0$, and replace its unavailable loss by the gradient-anchored legacy $q_i(x)=f_i(a_i)+g_i^{\top}(x-a_i)+(\kappa_i/2)\|x-a_i\|_2^2$. Its gradient is $\nabla q_i(x)=g_i+\kappa_i(x-a_i)$ and therefore $\nabla q_i(a_i)=\nabla f_i(a_i)$ exactly. The abstract states that when $f_i$ is $\mu_i$-strongly convex and $L_i$-smooth, the optimal legacy curvature is $\kappa_i=(\mu_i+L_i)/2$; here $\mu_i$ and $L_i$ bound the Hessian spectrum, $\mu_i I\preceq\nabla^2f_i(x)\preceq L_iI$. The quadratic Taylor-remainder bound is $|f_i(x)-f_i(a_i)-g_i^\top(x-a_i)|\le (L_i/2)\|x-a_i\|^2$, while choosing the midpoint curvature minimizes the worst-case curvature mismatch $\max\{|\kappa_i-\mu_i|,|L_i-\kappa_i|\}$. Multiple departed clients compose additively: $Q(x)=\sum_{i\in D}q_i(x)=C+b^\top x+(1/2)x^\top Hx$, where $H=\sum_{i\in D}\kappa_iI$, $b=\sum_{i\in D}(g_i-\kappa_i a_i)$, and $C$ is irrelevant to gradients. The retained legacy gradient is $\nabla Q(x)=b+Hx$.

## Key formulas

- $$F(x)=\sum_{i=1}^{n}f_i(x),\qquad x\in\mathbb{R}^{d}.$$
- $$q_i(x)=f_i(a_i)+g_i^{\top}(x-a_i)+\frac{\kappa_i}{2}\|x-a_i\|_2^2,\qquad g_i=\nabla f_i(a_i).$$
- $$\nabla q_i(x)=g_i+\kappa_i(x-a_i),\qquad \nabla q_i(a_i)=g_i=\nabla f_i(a_i),\qquad \kappa_i=\frac{\mu_i+L_i}{2}.$$
- $$Q(x)=\sum_{i\in D}q_i(x)=C+b^{\top}x+\frac12x^{\top}Hx,\quad H=\sum_{i\in D}\kappa_iI,\quad b=\sum_{i\in D}(g_i-\kappa_i a_i),\quad \nabla Q(x)=b+Hx.$$

## Implementation notes

(1) Integrate this at the federated-server or decentralized gradient-tracking objective layer. Each active client normally contributes a minibatch gradient to the shared model $x$. On planned or detected departure, the client sends $(a_i,g_i,\kappa_i)$, where $a_i$ is the current global parameter vector and $g_i$ is a final minibatch or EMA gradient. Do not retain the full client model history or data. The server maintains the aggregate legacy state $(b,H)$; for isotropic curvature, store scalar $h=\sum_i\kappa_i$ and vector $b$, so the memory cost is one model-sized vector plus one scalar. (2) Pseudocode: `legacy_b=0; legacy_h=0`; on departure compute `a=x.detach()`, `g=client_gradient(a)`, estimate `mu,L`, set `kappa=(mu+L)/2`, then update `legacy_b += g-kappa*a; legacy_h += kappa`; at every subsequent server step compute `legacy_grad = legacy_b + legacy_h*x`, combine it with active-client gradient `g_active` as `g_total=g_active+legacy_grad`, and pass `g_total` to the existing optimizer or gradient-tracking update. If clients have unequal weights $w_i$, use `legacy_b += w_i*(g-kappa*a)` and `legacy_h += w_i*kappa`. (3) The exact mathematical operations are the gradient-matching quadratic and additive composition. Estimate $\mu_i,L_i$ cheaply using several Hessian-vector power iterations, a diagonal curvature estimate, or fixed global bounds; if unavailable, clip $\kappa$ to a tested range and record the estimate. The constant $f_i(a_i)$ is unnecessary for optimization. (4) First experiment: use a 10-client non-IID CIFAR-10 federated MLP or small CNN, remove clients sequentially after 20%, 40%, and 60% of training, and compare FedAvg or decentralized gradient tracking with client drop-and-forget, quadratic legacies with fixed $\kappa$, and midpoint curvature. Measure accuracy on departed-client data, accuracy on active-client data, loss immediately after departure, communication bytes, and accuracy at equal rounds. The expected signal is reduced post-departure loss jumps and higher departed-client accuracy without retaining client data; test whether midpoint curvature beats fixed curvature and whether the legacy state adds less than one model vector per departed-client group.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: The gradient-anchored legacy was implemented and all required mechanism checks passed: maximum anchor-gradient error was 3.10e-14, midpoint curvature matched (mu+L)/2 across four sweeps, and observed error-scaling ratios were exactly 1.0. In the toy optimization, legacy retention reduced final departed-client loss from 111.45 to 64.67 and full loss from 176.54 to 145.37; fixed and midpoint curvature were indistinguishable because the synthetic spectra were narrow.

### Mechanism check

- Verdict: The gradient-anchored legacy was implemented and all required mechanism checks passed: maximum anchor-gradient error was 3.10e-14, midpoint curvature matched (mu+L)/2 across four sweeps, and observed error-scaling ratios were exactly 1.0. In the toy optimization, legacy retention reduced final departed-client loss from 111.45 to 64.67 and full loss from 176.54 to 145.37; fixed and midpoint curvature were indistinguishable because the synthetic spectra were narrow.
- Confidence: 9/10
- Limitations: Only synthetic diagonal quadratic clients were tested with deterministic full-batch gradients and exact curvature bounds. CIFAR/FedAvg, stochastic gradients, communication timing, unequal weights, broader curvature heterogeneity, and multi-seed significance were not tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/292/bench_report.json)
- [legacy_experiment.py](https://synthcore.org/code/292/legacy_experiment.py)
- [quadratic_legacies_bench.py](https://synthcore.org/code/292/quadratic_legacies_bench.py)
- [report.md](https://synthcore.org/code/292/report.md)
- [report_bench_2026-08-31T002220.md](https://synthcore.org/code/292/report_bench_2026-08-31T002220.md)
- [results.json](https://synthcore.org/code/292/results.json)
- [Download all files as ZIP](https://synthcore.org/download/292)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.