Spectral-Guided Discounted Decentralized Optimizer

Integrate this into the federated or decentralized optimizer update immediately after each client receives a new streaming batch and before the next batch arrives. Store one model vector w_n per client, stacked as X in R^{N x d}; do not average models only once at the end. At round t, each client computes a gradient on its new batch, optionally augmented with a small local replay buffer of old examples. For k=0,...,K-1, execute: (1) each client computes g_n, the gradient of its exponentially weighted local surrogate; (2) communicate with graph neighbors and compute X <- M X - eta G, where row n of G is g_n; (3) continue until K iterations are complete. Use rho as a configurable forgetting factor. For a strict finite window, maintain W recent mini-batches or sufficient statistics and replace the exponential recursion by uniform weighting over the last W batches. Estimate lambda=max(|lambda_2(M)|,|lambda_N(M)|) once using power iteration restricted to the subspace orthogonal to 1_N. Estimate L with a short Hessian-vector power iteration or use a conservative backtracking estimate. Compute q=lambda+eta L, clamp eta below (1-lambda)/L, and choose K as the smallest integer satisfying q^K<=epsilon_consensus, subject to the communication budget. The mathematical recursion, consensus contraction, and predicted uniform-versus-discounted tracking behavior come from the paper; actual disagreement and curvature estimates are measured empirically. Track D_t=(1/N)sum_n||w_n-w_bar||^2, recent-window loss, current-batch loss, and recovery time after distribution shifts. First test on 10 non-IID clients using rotated-MNIST or sequential CIFAR-10 with a 2-layer MLP. Compare FedAvg/local SGD, fixed-K decentralized SGD, uniform weighting, rho in {0.9,0.99}, and windows W in {10,50}, keeping total gradient evaluations and communication rounds identical. Success means faster recovery after shifts and lower recent-window loss at equal communication, with disagreement remaining bounded. Also verify the predicted tradeoff: rho closer to one or larger W should reduce variance but increase tracking lag, while larger K should reduce disagreement and the decentralization bias.