# Pole-radius tuning for gradient tracking

- ID: 1324
- Canonical URL: https://synthcore.org/idea/1324/pole-radius-tuning-for-gradient-tracking
- API JSON: https://synthcore.org/api/idea/1324.json
- API Markdown: https://synthcore.org/api/idea/1324.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.23601](https://arxiv.org/abs/2607.23601)
- Category: optimization
- Solves: speedup, stability
- ML areas: optimizer, federated, training-dynamics
- Math tags: dynamical-systems, control-theory, spectral-theory, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace generic learning-rate selection in decentralized or federated gradient tracking with a low-dimensional minimax search over the exact scalar-mode pole radius. The optimizer chooses the step size that minimizes the worst predicted contraction over the observed graph spectrum and an estimated curvature interval, rather than relying only on conservative global bounds.

## Mathematical statement

The paper diagonalizes the symmetric communication matrix W, whose eigenvalues are 1=\lambda_1,\lambda_2,\ldots,\lambda_N, and analyzes each disagreement mode separately. For DIGing, the mode transfer function is g_i^{DG}(z)=-\alpha(z-1)/(z-\lambda_i)^2 for i\ge2, where \alpha is the step size and z is the discrete-time pole variable. For a quadratic local curvature q\in[\mu,L], the disagreement state matrix is M_{DG}(\lambda,q,\alpha)=\begin{bmatrix}\lambda&-\alpha\\q(\lambda-1)&\lambda-q\alpha\end{bmatrix}, giving the characteristic polynomial p_{DG}(r;\lambda,q,\alpha)=(r-\lambda)^2+q\alpha(r-1). Here r is a closed-loop pole, \lambda is a non-consensus communication eigenvalue, q is a scalar Hessian eigenvalue, and \mu,L are lower and upper curvature estimates. The predicted quadratic worst-case rate is \rho(\alpha)=\max_{\lambda\in\Lambda_{dis},q\in[\mu,L]}\max_{r:p_{DG}(r;\lambda,q,\alpha)=0}|r|, where \Lambda_{dis}=\{\lambda_2,\ldots,\lambda_N\}.

## Key formulas

- $$g_i^{DG}(z)=-\frac{\alpha(z-1)}{(z-\lambda_i)^2},\qquad i=2,\ldots,N.$$
- $$p_{DG}(r;\lambda,q,\alpha)=(r-\lambda)^2+q\alpha(r-1)=0.$$
- $$\alpha^*=\arg\min_{\alpha>0}\rho(\alpha),\qquad \rho(\alpha)=\max_{\lambda\in\Lambda_{dis},\,q\in[\mu,L]}\max_{r:p_{DG}(r;\lambda,q,\alpha)=0}|r|.$$

## Implementation notes

(1) Integrate this into the optimizer wrapper used by decentralized data-parallel training. Worker i stores parameters x_i and a gradient tracker y_i. Use DIGing updates x_i[k+1]=\sum_jW_{ij}x_j[k]-\alpha y_i[k] and y_i[k+1]=\sum_jW_{ij}y_j[k]+g_i[k+1]-g_i[k], with g_i[k]=
abla f_i(x_i[k]). (2) At initialization and every calibration window, estimate curvature. Use Hessian-vector power iteration on a representative minibatch for L; estimate \mu from the lower quantiles of gradient-difference Rayleigh quotients q_t=\langle g_t-g_{t-1},x_t-x_{t-1}\rangle/\|x_t-x_{t-1}\|^2, clipped to a positive floor. Estimate the non-consensus eigenvalues of W with eigendecomposition for small worker counts or Lanczos for larger graphs. (3) For 50-100 candidate step sizes, solve p_DG(r;\lambda,q,\alpha)=0 for every estimated \lambda and for q on a grid spanning [\mu,L]. Compute the maximum root modulus and select the stable candidate with minimum predicted \rho. Apply it for the next window; backtrack if tracker disagreement or loss increases sharply. The polynomial and root calculation come from the paper; curvature and graph spectra are empirical estimates. (4) First experiment: train ResNet-18 on CIFAR-10 with 4 or 8 workers using ring and expander graphs. Compare fixed-step DIGing, tuned DIGing, and centralized SGD at equal communication rounds and FLOPs. Measure loss versus synchronization rounds, final accuracy, tracker norm, and calibration overhead. Success means faster loss descent or higher accuracy at equal communication, with fewer divergence events.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built and ran an exact scalar-mode pole-radius calculator plus a deterministic quadratic DIGing experiment. The mechanism predictions were confirmed: the observed stability transition matched alpha_c=(1+lambda)^2/(2q) within sweep resolution, and the critical step scaled as 1/q with relative error below 0.04%. Minimax pole-radius search predicted a lower disagreement radius (0.868 versus 0.955 for the fixed step), but the selected very small step did not improve finite-horizon total optimization loss because the consensus/optimization mode was omitted from the disagreement-only objective; this validates the mechanism, not an end-to-end training win.

### Mechanism check

- Verdict: Built and ran an exact scalar-mode pole-radius calculator plus a deterministic quadratic DIGing experiment. The mechanism predictions were confirmed: the observed stability transition matched alpha_c=(1+lambda)^2/(2q) within sweep resolution, and the critical step scaled as 1/q with relative error below 0.04%. Minimax pole-radius search predicted a lower disagreement radius (0.868 versus 0.955 for the fixed step), but the selected very small step did not improve finite-horizon total optimization loss because the consensus/optimization mode was omitted from the disagreement-only objective; this validates the mechanism, not an end-to-end training win.
- Confidence: 8/10
- Limitations: Only small synthetic quadratic objectives and a four-worker symmetric graph were tested; no CIFAR-10, neural network, ring/expander comparison, curvature estimation, calibration overhead, backtracking, FLOP accounting, or centralized SGD baseline was implemented. The minimax objective used only disagreement modes, so it can favor alpha approaching zero when all non-consensus eigenvalues are positive; a practical implementation should include the consensus optimization rate or impose a lower step-size constraint.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/521/bench_report.json)
- [pole_radius_experiment.py](https://synthcore.org/code/521/pole_radius_experiment.py)
- [report.md](https://synthcore.org/code/521/report.md)
- [report_bench_2026-08-31T164543.md](https://synthcore.org/code/521/report_bench_2026-08-31T164543.md)
- [results.json](https://synthcore.org/code/521/results.json)
- [stage2_bench.py](https://synthcore.org/code/521/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/521)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.