✗ Failed on benchmark 2026

Pole-radius tuning for gradient tracking

Implementation & benchmark of arXiv:2607.23601 — Exact Worst-case Convergence Rates of Distributed Gradient Tracking Methods

Usefulness7/10

Difficulty5/10

Novelty6/10

Optimization Speedup Stability Optimizer Federated Training dynamics Dynamical systems Control theory Spectral theory Optimization

Source paper: Exact Worst-case Convergence Rates of Distributed Gradient Tracking Methods arXiv:2607.23601 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace generic learning-rate selection in decentralized or federated gradient tracking with a low-dimensional minimax search over the exact scalar-mode pole radius. The optimizer chooses the step size that minimizes the worst predicted contraction over the observed graph spectrum and an estimated curvature interval, rather than relying only on conservative global bounds.

Formulas

$$g_i^{DG}(z)=-\frac{\alpha(z-1)}{(z-\lambda_i)^2},\qquad i=2,\ldots,N.$$

$$p_{DG}(r;\lambda,q,\alpha)=(r-\lambda)^2+q\alpha(r-1)=0.$$

$$\alpha^*=\arg\min_{\alpha>0}\rho(\alpha),\qquad \rho(\alpha)=\max_{\lambda\in\Lambda_{dis},\,q\in[\mu,L]}\max_{r:p_{DG}(r;\lambda,q,\alpha)=0}|r|.$$

Mathematical statement

The paper diagonalizes the symmetric communication matrix W, whose eigenvalues are 1=\lambda_1,\lambda_2,\ldots,\lambda_N, and analyzes each disagreement mode separately. For DIGing, the mode transfer function is g_i^{DG}(z)=-\alpha(z-1)/(z-\lambda_i)^2 for i\ge2, where \alpha is the step size and z is the discrete-time pole variable. For a quadratic local curvature q\in[\mu,L], the disagreement state matrix is M_{DG}(\lambda,q,\alpha)=\begin{bmatrix}\lambda&-\alpha\\q(\lambda-1)&\lambda-q\alpha\end{bmatrix}, giving the characteristic polynomial p_{DG}(r;\lambda,q,\alpha)=(r-\lambda)^2+q\alpha(r-1). Here r is a closed-loop pole, \lambda is a non-consensus communication eigenvalue, q is a scalar Hessian eigenvalue, and \mu,L are lower and upper curvature estimates. The predicted quadratic worst-case rate is \rho(\alpha)=\max_{\lambda\in\Lambda_{dis},q\in[\mu,L]}\max_{r:p_{DG}(r;\lambda,q,\alpha)=0}|r|, where \Lambda_{dis}=\{\lambda_2,\ldots,\lambda_N\}.

Implementation notes

(1) Integrate this into the optimizer wrapper used by decentralized data-parallel training. Worker i stores parameters x_i and a gradient tracker y_i. Use DIGing updates x_i[k+1]=\sum_jW_{ij}x_j[k]-\alpha y_i[k] and y_i[k+1]=\sum_jW_{ij}y_j[k]+g_i[k+1]-g_i[k], with g_i[k]=
abla f_i(x_i[k]). (2) At initialization and every calibration window, estimate curvature. Use Hessian-vector power iteration on a representative minibatch for L; estimate \mu from the lower quantiles of gradient-difference Rayleigh quotients q_t=\langle g_t-g_{t-1},x_t-x_{t-1}\rangle/\|x_t-x_{t-1}\|^2, clipped to a positive floor. Estimate the non-consensus eigenvalues of W with eigendecomposition for small worker counts or Lanczos for larger graphs. (3) For 50-100 candidate step sizes, solve p_DG(r;\lambda,q,\alpha)=0 for every estimated \lambda and for q on a grid spanning [\mu,L]. Compute the maximum root modulus and select the stable candidate with minimum predicted \rho. Apply it for the next window; backtrack if tracker disagreement or loss increases sharply. The polynomial and root calculation come from the paper; curvature and graph spectra are empirical estimates. (4) First experiment: train ResNet-18 on CIFAR-10 with 4 or 8 workers using ring and expander graphs. Compare fixed-step DIGing, tuned DIGing, and centralized SGD at equal communication rounds and FLOPs. Measure loss versus synchronization rounds, final accuracy, tracker norm, and calibration overhead. Success means faster loss descent or higher accuracy at equal communication, with fewer divergence events.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built and ran an exact scalar-mode pole-radius calculator plus a deterministic quadratic DIGing experiment. The mechanism predictions were confirmed: the observed stability transition matched alpha_c=(1+lambda)^2/(2q) within sweep resolution, and the critical step scaled as 1/q with relative error below 0.04%. Minimax pole-radius search predicted a lower disagreement radius (0.868 versus 0.955 for the fixed step), but the selected very small step did not improve finite-horizon total optimization loss because the consensus/optimization mode was omitted from the disagreement-only objective; this validates the mechanism, not an end-to-end training win.

Agent confidence: 8/10

Baseline: Fixed DIGing alpha=0.05: predicted disagreement rho=0.9555; final quadratic loss=5.6540; final mean parameter error=2.09e-9 after 250 rounds.
Idea: Pole-radius tuned alpha=0.0005: predicted disagreement rho=0.8682; final quadratic loss=6.0573; final mean parameter error=0.5416 after 250 rounds. Stability boundary sweep gave rho=0.9933 at 0.995 alpha_c and 1.0067 at 1.005 alpha_c. Inverse-curvature boundary relative error was 0.004%–0.039%.

Limitations: Only small synthetic quadratic objectives and a four-worker symmetric graph were tested; no CIFAR-10, neural network, ring/expander comparison, curvature estimation, calibration overhead, backtracking, FLOP accounting, or centralized SGD baseline was implemented. The minimax objective used only disagreement modes, so it can favor alpha approaching zero when all non-consensus eigenvalues are positive; a practical implementation should include the consensus optimization rate or impose a lower step-size constraint.

How to run: python3 pole_radius_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.23601) Sep 2, 2026

Significantly WORSE than the tuned baseline (+812.7%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 22.9948
Idea mean: 209.8789
Effect (Δ): 186.8841 (+812.7%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±41.9%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the registered tabular/Friedman track was tested, using a four-worker ring and 24 training epochs. Larger graphs, expander communication, curvature power iteration, backtracking, centralized SGD, communication/FLOP accounting, and the other built-in tracks were not tested.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

The pole-radius DIGing variant was implemented and evaluated on the registered tabular Friedman regression track with the shared mlp_tiny architecture, four-worker ring communication, an equal learning-rate grid, and 8 paired seeds. The baseline mean test MSE was 22.9948 versus 209.8789 for the idea; paired delta was +186.8841 with permutation p=0.0081 and zero idea wins, so the idea was significantly worse. The trained-model mechanism signature was not confirmed because the predicted pole radius (0.5111) did not match the observed tracker-disagreement ratio (0.9516).

Baseline: best lr=0.003; full mean test MSE=22.994831800460815; std=11.385712752026535
Idea: pole-radius tuning selected alpha=0.001; full mean test MSE=209.8788948059082; std=12.027703233030866

Benchmark result