Unverified 2026

Phase-Repulsive Worker Throttling

Implementation & benchmark of arXiv:2607.19638 — Do Co-Located AI Training Jobs Synchronize? Load-Dependent Throttling as a Coupling Mechanism for Phase-Locking Behind a Shared Power Cap

Usefulness6/10

Difficulty6/10

Novelty7/10

Dynamics Stability Scalability Speedup Optimizer Training dynamics Scheduler Dynamical systems Bifurcations Control theory Statistics

Source paper: Do Co-Located AI Training Jobs Synchronize? Load-Dependent Throttling as a Coupling Mechanism for Phase-Locking Behind a Shared Power Cap arXiv:2607.19638 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Treat periodic update bursts from distributed training workers or parameter blocks as oscillator phases, and use a shared adaptive compute or learning-rate cap to create deliberately phase-repulsive coupling. When aggregate demand is high, throttle workers currently near their compute peak and preferentially release workers in low-demand phases, spreading communication and gradient-update bursts instead of allowing them to lock together. The controller should be disabled or retuned when its measured feedback delay crosses the phase-lag boundary at which effective coupling changes sign.

Formulas

$$\dot{\theta}_i=\omega_i+\frac{K_1}{N}\sum_{j=1}^{N}\sin(\theta_j-\theta_i+\Phi),\qquad i=1,\ldots,N.$$

$$r(t)e^{\mathrm{i}\psi(t)}=\frac{1}{N}\sum_{j=1}^{N}e^{\mathrm{i}\theta_j(t)},\qquad K_{\mathrm{sync}}=K_1\cos\Phi.$$

$$K_{\mathrm{sync}}(\tau)=K_1\cos\bigl(\Phi_0+\Omega\tau\bigr),\qquad K_{\mathrm{sync},c}=\frac{2}{\pi g(\omega_0)}.$$

$$u_i(t+1)=\operatorname{clip}\left(u_0\left[1-\alpha\bigl(p(t)-\bar p(t)\bigr)\sin\bigl(\theta_i(t)-\psi(t)\bigr)\right],u_{\min},u_{\max}\right).$$

Mathematical statement

For worker phases $\theta_i$, intrinsic iteration frequencies $\omega_i$, and coupling strength $K_1$, use $\dot{\theta}_i=\omega_i+(K_1/N)\sum_j\sin(\theta_j-\theta_i+\Phi)$. Coherence is measured by $r e^{\mathrm{i}\psi}=N^{-1}\sum_j e^{\mathrm{i}\theta_j}$, where $r=0$ denotes dispersed phases and $r=1$ perfect locking. Let $\Omega$ be the dominant worker-cycle frequency, $\tau$ the control delay, and $\Phi_0$ the phase shift from the workload waveform and controller. The synchronizing component is $K_{\mathrm{sync}}=K_1\cos(\Phi_0+\Omega\tau)$; its sign changes when the total lag reaches a quarter cycle modulo $\pi$. For a unimodal natural-frequency density $g(\omega)$ centered at $\omega_0$, the incoherent state loses stability near $K_{\mathrm{sync},c}=2/[\pi g(\omega_0)]$. The neural controller applies a phase-dependent per-worker resource multiplier, with saturation kept inactive during the small-signal regime.

Implementation notes

1. Integration point: instrument each data-parallel worker, pipeline stage, or selected parameter block with a scalar phase. Define a worker phase by band-pass filtering its recent GPU-power, step-time, gradient-norm, or all-reduce-traffic trace and applying a Hilbert transform; a simpler prototype can assign phase zero at each observed compute-burst onset and linearly advance it between bursts. Let $p_t$ be normalized aggregate demand and $\bar p_t$ its low-pass average. Let $u_i$ be a worker's microbatch allowance, token budget, local learning-rate multiplier, or compute quota. 2. Pseudocode: at every control interval, compute $z=N^{-1}\sum_i e^{\mathrm{i}\theta_i}$, $r=|z|$, and $\psi=\arg z$. Estimate the dominant frequency $\Omega$ using an autocorrelation peak or FFT, estimate delay $\tau$ from the cross-correlation between demand and applied throttling, and calculate $K_{\mathrm{sync}}=K_1\cos(\Phi_0+\Omega\tau)$. If $K_{\mathrm{sync}}<0$, update each quota with $u_i\leftarrow\operatorname{clip}(u_0[1-\alpha(p_t-\bar p_t)\sin(\theta_i-\psi)],u_{\min},u_{\max})$. If it is positive, reduce $\alpha$, compensate for delay, or disable the controller. 3. Computed versus estimated: $r$, $\psi$, $\Omega$, and $\tau$ are estimated online. Calibrate $K_1$ and $\Phi_0$ by injecting a small sinusoidal quota modulation and fitting the first-harmonic amplitude and phase of aggregate demand. The Kuramoto threshold is a small-signal prediction, not a formal theorem for stochastic NN training. 4. First cheap experiment: run 8 asynchronous PyTorch workers training ResNet-18 on CIFAR-10 with deliberately synchronized 20-step compute/communication bursts. Compare no controller, conventional global token throttling, and this controller over several artificial delays $\tau$. Measure phase order parameter $r$, all-reduce peak-to-mean traffic, step-time variance, throughput, and validation accuracy. The predicted signature is a sharp change from decreasing to increasing $r$ when $\Phi_0+\Omega\tau$ crosses a zero of the cosine, followed by coherence growth when measured $K_{\mathrm{sync}}$ exceeds approximately $2/[\pi g(\omega_0)]$. The empirical transition should agree with the calibrated prediction within about 20%.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.