# Asymptotic Training-Horizon Controller - ID: 1312 - Canonical URL: https://synthcore.org/idea/1312/asymptotic-training-horizon-controller - API JSON: https://synthcore.org/api/idea/1312.json - API Markdown: https://synthcore.org/api/idea/1312.md - Verification status: unverified - Source: [arXiv:2607.23287](https://arxiv.org/abs/2607.23287) - Category: training - Solves: speedup, stability - ML areas: training-dynamics, scheduler, fine-tuning - Math tags: approximation-theory, statistics, optimization, numerical-analysis - Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10 ## Idea description Model a checkpointed validation metric as a finite asymptotic expansion in known decay features, such as powers of training step, and estimate its limiting value using sliding least squares. Use a ridge-stabilized fit and require agreement across multiple windows before stopping, preventing the controller from reacting to transient non-asymptotic behavior. ## Mathematical statement The paper studies sequences with an asymptotic form whose unknown coefficients are recovered by linear least squares over a sliding window. For a scalar metric y(t), use the adapted expansion y(t)=theta_0+sum_{j=1}^m theta_j phi_j(t)+epsilon(t), where t is the checkpoint index, theta_0 is the limiting metric, phi_j(t) are fixed decay features such as t^{-p_j}, and epsilon(t) is the unmodeled remainder. Given the window W_t={t-w+1,...,t}, sliding Tikhonov least squares is hat(theta)_t=argmin_theta sum_{i in W_t}(y_i-phi(i)^T theta)^2+lambda||D theta||_2^2. Here phi(i)=[1,phi_1(i),...,phi_m(i)]^T, D is a diagonal matrix that can leave the intercept unpenalized, and lambda>0 controls ill-conditioning. The fitted asymptotic limit is hat(y)_infinity=hat(theta)_{0,t}. The paper's rate example gives lim_{n to infinity}|r_n-alpha_3|sqrt(n)=2|alpha_3|, equivalently r_n-alpha_3=Theta(n^{-1/2}); in the neural adaptation, analogous residual improvement and decay-rate estimates are used only when several windows agree, because the paper explicitly identifies slow convergence and divergence as possible failure modes. ## Key formulas - $$\lim_{n\to\infty}\left\lvert r_n-\alpha_{3}\right\rvert\sqrt{n}=2\left\lvert\alpha_{3}\right\rvert=2\alpha_{3}\quad\Longrightarrow\quad r_n-\alpha_{3}=\Theta\left(\frac{1}{\sqrt{n}}\right)=o(1).$$ - $$y(t)=\theta_0+\sum_{j=1}^{m}\theta_j t^{-p_j}+\varepsilon(t),\qquad \phi(t)=\left[1,t^{-p_1},\ldots,t^{-p_m}\right]^\top.$$ - $$\widehat{\theta}_t=\arg\min_{\theta}\sum_{i=t-w+1}^{t}\left(y_i-\phi(i)^\top\theta\right)^2+\lambda\left\|D\theta\right\|_2^2.$$ - $$\widehat{y}_{\infty,t}=\widehat{\theta}_{0,t},\qquad \widehat{\Delta}_{t,H}=\widehat{y}(t)-\widehat{y}(t+H)=\widehat{\theta}_t^\top\left[\phi(t)-\phi(t+H)\right].$$ ## Implementation notes Integrate this at the training loop's checkpoint and evaluation path, not inside the forward or backward pass. Every K optimizer steps, record a smoothed validation loss y_i, the checkpoint index t_i, and optionally training loss and gradient norm as separate series. Choose a small candidate basis, for example phi(t)=[1,t^{-1/2},t^{-1},t^{-3/2}], after rescaling t by the first checkpoint to avoid numerical magnitude problems. For each new checkpoint and each window size w in {8,16,32}, solve the ridge system (X^T X+lambda D^T D)theta=X^T y, with D=diag(0,1,1,1), lambda between 1e-4 and 1e-2, and compute y_infinity=theta[0]. Estimate expected improvement over H future checkpoints with theta^T(phi(t)-phi(t+H)). Compute a stability score from the standard deviation of y_infinity across window sizes and from the fit residual; do not stop unless both are below thresholds for three consecutive evaluations. Pseudocode: record y; for each w construct X and solve the ridge system; obtain y_infinity,w and delta_w; if median(delta_w)