✗ Failed on benchmark 2026

Quadratic Client Legacies

Implementation & benchmark of arXiv:2607.09939 — Robust Decentralized Optimization under Node Failures via Adaptive Regularization

Usefulness7/10

Difficulty4/10

Novelty7/10

Training Stability Sample efficiency Scalability Optimizer Training Fine tuning Optimization Convex analysis Dynamical systems

Source paper: Robust Decentralized Optimization under Node Failures via Adaptive Regularization arXiv:2607.09939 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

When a federated or decentralized client leaves, transmit a small gradient-anchored quadratic surrogate instead of discarding its loss. The surrogate preserves the client's gradient exactly at the departure model and supplies a controlled approximation away from that point, allowing training to retain information from unavailable clients with constant memory and communication.

Formulas

$$F(x)=\sum_{i=1}^{n}f_i(x),\qquad x\in\mathbb{R}^{d}.$$

$$q_i(x)=f_i(a_i)+g_i^{\top}(x-a_i)+\frac{\kappa_i}{2}\|x-a_i\|_2^2,\qquad g_i=\nabla f_i(a_i).$$

$$\nabla q_i(x)=g_i+\kappa_i(x-a_i),\qquad \nabla q_i(a_i)=g_i=\nabla f_i(a_i),\qquad \kappa_i=\frac{\mu_i+L_i}{2}.$$

$$Q(x)=\sum_{i\in D}q_i(x)=C+b^{\top}x+\frac12x^{\top}Hx,\quad H=\sum_{i\in D}\kappa_iI,\quad b=\sum_{i\in D}(g_i-\kappa_i a_i),\quad \nabla Q(x)=b+Hx.$$

Mathematical statement

The paper's optimization problem is $F(x)=\sum_{i=1}^{n}f_i(x)$, where $x\in\mathbb{R}^d$ is the shared model and $f_i$ is client $i$'s private loss. For a departing client $i$, choose an anchor $a_i$ equal to the shared parameters at departure, gradient $g_i=\nabla f_i(a_i)$, and quadratic curvature $\kappa_i>0$, and replace its unavailable loss by the gradient-anchored legacy $q_i(x)=f_i(a_i)+g_i^{\top}(x-a_i)+(\kappa_i/2)\|x-a_i\|_2^2$. Its gradient is $\nabla q_i(x)=g_i+\kappa_i(x-a_i)$ and therefore $\nabla q_i(a_i)=\nabla f_i(a_i)$ exactly. The abstract states that when $f_i$ is $\mu_i$-strongly convex and $L_i$-smooth, the optimal legacy curvature is $\kappa_i=(\mu_i+L_i)/2$; here $\mu_i$ and $L_i$ bound the Hessian spectrum, $\mu_i I\preceq\nabla^2f_i(x)\preceq L_iI$. The quadratic Taylor-remainder bound is $|f_i(x)-f_i(a_i)-g_i^\top(x-a_i)|\le (L_i/2)\|x-a_i\|^2$, while choosing the midpoint curvature minimizes the worst-case curvature mismatch $\max\{|\kappa_i-\mu_i|,|L_i-\kappa_i|\}$. Multiple departed clients compose additively: $Q(x)=\sum_{i\in D}q_i(x)=C+b^\top x+(1/2)x^\top Hx$, where $H=\sum_{i\in D}\kappa_iI$, $b=\sum_{i\in D}(g_i-\kappa_i a_i)$, and $C$ is irrelevant to gradients. The retained legacy gradient is $\nabla Q(x)=b+Hx$.

Implementation notes

(1) Integrate this at the federated-server or decentralized gradient-tracking objective layer. Each active client normally contributes a minibatch gradient to the shared model $x$. On planned or detected departure, the client sends $(a_i,g_i,\kappa_i)$, where $a_i$ is the current global parameter vector and $g_i$ is a final minibatch or EMA gradient. Do not retain the full client model history or data. The server maintains the aggregate legacy state $(b,H)$; for isotropic curvature, store scalar $h=\sum_i\kappa_i$ and vector $b$, so the memory cost is one model-sized vector plus one scalar. (2) Pseudocode: legacy_b=0; legacy_h=0; on departure compute a=x.detach(), g=client_gradient(a), estimate mu,L, set kappa=(mu+L)/2, then update legacy_b += g-kappa*a; legacy_h += kappa; at every subsequent server step compute legacy_grad = legacy_b + legacy_h*x, combine it with active-client gradient g_active as g_total=g_active+legacy_grad, and pass g_total to the existing optimizer or gradient-tracking update. If clients have unequal weights $w_i$, use legacy_b += w_i*(g-kappa*a) and legacy_h += w_i*kappa. (3) The exact mathematical operations are the gradient-matching quadratic and additive composition. Estimate $\mu_i,L_i$ cheaply using several Hessian-vector power iterations, a diagonal curvature estimate, or fixed global bounds; if unavailable, clip $\kappa$ to a tested range and record the estimate. The constant $f_i(a_i)$ is unnecessary for optimization. (4) First experiment: use a 10-client non-IID CIFAR-10 federated MLP or small CNN, remove clients sequentially after 20%, 40%, and 60% of training, and compare FedAvg or decentralized gradient tracking with client drop-and-forget, quadratic legacies with fixed $\kappa$, and midpoint curvature. Measure accuracy on departed-client data, accuracy on active-client data, loss immediately after departure, communication bytes, and accuracy at equal rounds. The expected signal is reduced post-departure loss jumps and higher departed-client accuracy without retaining client data; test whether midpoint curvature beats fixed curvature and whether the legacy state adds less than one model vector per departed-client group.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

The gradient-anchored legacy was implemented and all required mechanism checks passed: maximum anchor-gradient error was 3.10e-14, midpoint curvature matched (mu+L)/2 across four sweeps, and observed error-scaling ratios were exactly 1.0. In the toy optimization, legacy retention reduced final departed-client loss from 111.45 to 64.67 and full loss from 176.54 to 145.37; fixed and midpoint curvature were indistinguishable because the synthetic spectra were narrow.

Agent confidence: 9/10

Baseline: Drop-and-forget at step 200: full loss 176.5402, departed loss 111.4544, active loss 65.0857.
Idea: Quadratic legacy at step 200: full loss 145.3678, departed loss 64.6700, active loss 80.6978; anchor error 3.10e-14; error-scaling ratio 1.0.

Limitations: Only synthetic diagonal quadratic clients were tested with deterministic full-batch gradients and exact curvature bounds. CIFAR/FedAvg, stochastic gradients, communication timing, unequal weights, broader curvature heterogeneity, and multi-seed significance were not tested.

How to run: python3 legacy_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.09939) Sep 2, 2026

Significantly WORSE than the tuned baseline (+28.1%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.7908
Idea mean: 9.9803
Effect (Δ): 2.1895 (+28.1%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 0 configs

Limitations:

Only the tabular Friedman regression track was tested. The experiment simulated one departure at epoch 8 with four fixed private partitions, isotropic fixed curvature, and one client update per epoch; multiple departures, unequal client weights, availability schedules, and communication costs were not tested.

How to run: python3 quadratic_legacies_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Built and ran quadratic client legacies on the structurally matched tabular Friedman regression track using paired seeds, the same mlp_tiny architecture, and equal Adam budgets. Baseline mean test MSE was 7.7908 versus 9.9803 for the idea, with paired delta +2.1895 and permutation p=0.0081, so the idea was significantly worse. The trained-model gradient signature failed as well: predicted legacy versus observed departed-client gradient cosine was -0.8786.

Baseline: mean test MSE 7.7907896638, std 0.7662284654, best lr=0.01, 8 paired seeds
Idea: mean test MSE 9.9802609682, std 1.2184810728, best lr=0.01 and kappa=0.1, paired delta +2.1894713044, p=0.0081, 0/8 wins
Mechanism signature: anchor_gradient_error=0.0; predicted_q_gradient_norm=59.0085; observed_departed_gradient_norm=44.9906; cosine=-0.878625; confirmed=false

Benchmark result