# Impedance-Calibrated Learning-Rate Control

- ID: 3060
- Canonical URL: https://synthcore.org/idea/3060/impedance-calibrated-learning-rate-control
- API JSON: https://synthcore.org/api/idea/3060.json
- API Markdown: https://synthcore.org/api/idea/3060.md
- Verification status: mechanism_failed
- Source: [arXiv:2609.02458](https://arxiv.org/abs/2609.02458)
- Category: dynamics
- Solves: stability, speedup
- ML areas: optimizer, scheduler, training-dynamics
- Math tags: dynamical-systems, control-theory, spectral-theory, linear-algebra
- Ratings: usefulness 8/10; difficulty 5/10; novelty 7/10

## Idea description

Treat local neural-network training as a driven linear system and periodically modulate the learning rate by a small sinusoid. Estimate the transfer function from this modulation to loss or gradient observables, fit its relaxation poles, and set the learning rate below the measured instability boundary.

## Mathematical statement

Linearize training dynamics as \(\dot{x}=Ax+Bu\), where \(x\) is parameter displacement, \(A\) is the local training Jacobian, \(u\) is a scalar learning-rate modulation, and \(B\) is its input direction. For observable \(y=Cx\), with observation row vector \(C\), the frequency response is \(H(i\omega)=C(i\omega I-A)^{-1}B\). For real stable modes, \(H(i\omega)=\sum_k a_k/(1+i\omega\tau_k)\), where \(\tau_k=-1/\lambda_k>0\) are relaxation times, \(\lambda_k\) are stable eigenvalues, and \(a_k\) are residues. For a quadratic local loss, discrete gradient descent has mode update \(z_{k,t+1}=(1-\eta\lambda_k)z_{k,t}\), stable when \(0<\eta<2/\lambda_k\).

## Key formulas

- $$\dot{x}=Ax+Bu,\qquad y=Cx$$
- $$H(i\omega)=\frac{\widehat{y}(\omega)}{\widehat{u}(\omega)}=C(i\omega I-A)^{-1}B=\sum_{k=1}^{N}\frac{a_k}{1+i\omega\tau_k},\qquad \tau_k=-\lambda_k^{-1}>0$$
- $$z_{k,t+1}=(1-\eta\lambda_k)z_{k,t},\qquad |1-\eta\lambda_k|<1\Longleftrightarrow 0<\eta<\frac{2}{\lambda_k}$$
- $$\eta_{t+1}=\operatorname{clip}\left(\eta_t\min\left(1.1,\frac{2\gamma}{\eta_t\widehat{\lambda}_{\max}}\right),\eta_{\min},\eta_{\max}\right),\qquad 0<\gamma<1$$

## Implementation notes

Integrate this as a wrapper around SGD, initially avoiding Adam so the identified dynamics remain interpretable. Every K optimizer steps, apply a small learning-rate probe \(u_t=\epsilon\sin(\omega t)\) for several frequencies spanning approximately one decade below and above the estimated inverse relaxation time. Record minibatch loss, gradient norm, and optionally a random projection of parameter displacement. For each frequency, estimate the complex response with lock-in correlations, \(\widehat H(\omega)=2\langle y_t e^{-i\omega t}\rangle/(\epsilon T)\), over an integer number of probe periods. Fit one to four positive time constants and residues by nonlinear least squares. The paper-derived quantities are the response poles and RC-style decomposition; the Jacobian, residues, and effective curvatures are estimated empirically. Infer \(\widehat{\lambda}_{\max}=2/\widehat{\eta}_{\mathrm{crit}}\) from the largest learning rate whose oscillations still decay, then operate at \(\gamma\widehat{\eta}_{\mathrm{crit}}\), with \(\gamma=0.7\). First experiment: train a three-layer MLP on MNIST and compare fixed SGD, cosine decay, and impedance control at equal update count. Sweep fixed learning rates around the predicted boundary. The falsifiable signature is a sharp transition from decaying to growing or alternating loss oscillations near \(\eta_c\); the impedance estimate should predict \(\eta_c\) within 20 percent, while controlled training should remain below the boundary and reduce settling time.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a reproducible lock-in transfer-function sanity check, quadratic stability test, and small CUDA digits-MLP comparison in experiment.py. The core math was confirmed: corrected lock-in recovery had 0.82% relative error, and the quadratic transition occurred at the predicted eta=2/lambda_max=0.4. The training controller showed a promising result versus the selected fixed and cosine baselines, with test loss 0.0755 versus 0.1098 and 0.1616, but this is not a demonstrated impedance-control win because the implementation used Hessian power estimates rather than fitted sinusoidal poles/residues, and fixed-rate eta=0.6 performed better.

### Mechanism check

- Verdict: Built a reproducible lock-in transfer-function sanity check, quadratic stability test, and small CUDA digits-MLP comparison in experiment.py. The core math was confirmed: corrected lock-in recovery had 0.82% relative error, and the quadratic transition occurred at the predicted eta=2/lambda_max=0.4. The training controller showed a promising result versus the selected fixed and cosine baselines, with test loss 0.0755 versus 0.1098 and 0.1616, but this is not a demonstrated impedance-control win because the implementation used Hessian power estimates rather than fitted sinusoidal poles/residues, and fixed-rate eta=0.6 performed better.
- Confidence: 8/10
- Limitations: Only one random seed and one small sklearn digits dataset were tested. The real-training prototype does not implement the proposed multi-frequency probing, pole/residue nonlinear fit, or loss/gradient lock-in estimator; it estimates curvature directly with a short Hessian-vector power iteration. No FLOP-normalized or repeated-seed comparison, larger model, MNIST, or robust instability-boundary sweep was performed.

## Artifacts

- [experiment.py](https://synthcore.org/code/1214/experiment.py)
- [report.md](https://synthcore.org/code/1214/report.md)
- [results.json](https://synthcore.org/code/1214/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1214)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.