# Finite-Horizon Validation Boundary

- ID: 1761
- Canonical URL: https://synthcore.org/idea/1761/finite-horizon-validation-boundary
- API JSON: https://synthcore.org/api/idea/1761.json
- API Markdown: https://synthcore.org/api/idea/1761.md
- Verification status: unverified
- Source: [arXiv:2608.03889](https://arxiv.org/abs/2608.03889)
- Category: training
- Solves: accuracy, sample-efficiency, stability
- ML areas: training, training-dynamics, scheduler, loss
- Math tags: probability, statistics, stochastic-processes
- Ratings: usefulness 5/10; difficulty 4/10; novelty 5/10

## Idea description

Replace pointwise validation tests or infinite-horizon confidence sequences with a confidence horizon covering exactly the next H validation checks. Use the resulting simultaneous band to stop evaluating or stop training once the probability of further improvement falls below a target threshold, while spending less statistical slack than an anytime-valid method.

## Mathematical statement

The paper defines a classical confidence interval C_n for a parameter \mu by \lim_{n    o\infty}\mathsf{P}(\mu\in C_n)=0.95, and an asymptotically valid confidence sequence \{C_k^{(m)}\}_{k=m}^{\infty} by \lim_{m    o\infty}\mathsf{P}(\forall k\ge m,\ \mu\in C_k^{(m)})=0.95. A confidence horizon replaces the infinite index set with the finite planned set k\in\{m,\ldots,m+H\}. For a concrete neural implementation, let Y_i be the scalar validation-loss difference observed on independent or weakly dependent evaluation example i, with mean \mu, S_k=\sum_{i=1}^k(Y_i-\bar Y_k), and \bar Y_k=k^{-1}\sum_{i=1}^kY_i. Under a CLT and consistent variance estimate \hat\sigma_k^2, use C_k=[\bar Y_k-q_{H,\alpha}\hat\sigma_k/\sqrt{k},\ \bar Y_k+q_{H,\alpha}\hat\sigma_k/\sqrt{k}], where q_{H,\alpha} is chosen so that \mathsf{P}(\max_{1\le k\le H}|Z_k|\le q_{H,\alpha})=1-\alpha and (Z_1,\ldots,Z_H) is a zero-mean Gaussian vector with covariance \operatorname{Cov}(Z_i,Z_j)=\sqrt{\min(i,j)/\max(i,j)}. This covariance is the standardized partial-sum covariance. The finite maximum, rather than an infinite-time supremum, is the source of the sharper boundary.

## Key formulas

- $$\lim_{n\to\infty}\mathsf{P}(\mu\in C_n)=0.95.$$
- $$\lim_{m\to\infty}\mathsf{P}\left(\forall k\in\{m,m+1,m+2,\dots\},\ \mu\in C_k^{(m)}\right)=0.95.$$
- $$\mathsf{P}\left(\forall k\in\{m,\ldots,m+H\},\ \mu\in C_k^{(m,H)}\right)\ \longrightarrow\ 1-\alpha.$$
- $$C_k=\left[\bar Y_k-q_{H,\alpha}\frac{\hat\sigma_k}{\sqrt{k}},\ \bar Y_k+q_{H,\alpha}\frac{\hat\sigma_k}{\sqrt{k}}\right],\qquad \mathsf{P}\left(\max_{1\le k\le H}|Z_k|\le q_{H,\alpha}\right)=1-\alpha,\quad \operatorname{Cov}(Z_i,Z_j)=\sqrt{\frac{\min(i,j)}{\max(i,j)}}.$$

## Implementation notes

Integrate this into validation-based checkpoint selection, not into the forward pass. Fix a candidate checkpoint or a fixed incumbent-versus-candidate comparison, then evaluate both models on a stream of fresh validation examples or minibatches. Define Y_i=loss(incumbent,x_i)-loss(candidate,x_i), so positive \mu means that the candidate is better; use paired examples whenever possible to reduce variance. Before evaluation, choose H, the maximum number of allowed peeks, and \alpha, for example H=20 and \alpha=0.05. Precompute q_{H,\alpha} by drawing a large number of Gaussian vectors with covariance R_{ij}=sqrt(min(i,j)/max(i,j)), taking the empirical (1-alpha)-quantile of max_k |Z_k|. At evaluation step k, compute the running mean \bar Y_k and online variance \hat\sigma_k^2, then form C_k with the displayed finite-horizon formula. Pseudocode: initialize sums s=0, squares v=0; for k=1,...,H, draw validation minibatch B_k, set y_k=mean(loss_incumbent(B_k)-loss_candidate(B_k)), update s and v, compute mean=s/k and standard error=sqrt(v/(k-1))/sqrt(k), form [mean-q*se,mean+q*se]; accept the candidate if the lower endpoint exceeds zero, reject or stop evaluating if the upper endpoint is below a practical threshold \delta, otherwise continue. The mathematics supplies the simultaneous boundary; the variance and any minibatch-dependence correction are estimated empirically. First test on CIFAR-10 with a small ResNet and 20 sequential validation peeks, comparing pointwise 95% intervals, patience-based early stopping, and an infinite-horizon confidence sequence. Measure false accept/reject rates over repeated runs, validation examples consumed, and final test accuracy. Success is fewer validation batches or earlier reliable checkpoint decisions at the same error rate, with visibly narrower bands than the infinite-horizon baseline.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.