# Finite-Warp UCB for Neural Hyperparameter Search

- ID: 3118
- Canonical URL: https://synthcore.org/idea/3118/finite-warp-ucb-for-neural-hyperparameter-search
- API JSON: https://synthcore.org/api/idea/3118.json
- API Markdown: https://synthcore.org/api/idea/3118.md
- Verification status: unverified
- Source: [arXiv:2609.02993](https://arxiv.org/abs/2609.02993)
- Category: optimization
- Solves: sample-efficiency, accuracy, stability
- ML areas: fine-tuning, optimizer, scheduler
- Math tags: probability, information-theory, statistics, approximation-theory
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Replace a single-coordinate GP surrogate used for expensive neural-network hyperparameter trials with a finite bank of input-warped kernels. Each trial may select whichever warp currently gives the most promising upper-confidence acquisition, including after observing all previous results, while confidence parameters are enlarged to account for the finite library. This should recover the benefits of log-scaled or localized geometries without committing to one hand-designed parameterization.

## Mathematical statement

For a finite warp library indexed by \(\theta\in\Theta\), define \(k_\theta(x,x')=k_0(w_\theta(x),w_\theta(x'))\), where \(x\) is a neural hyperparameter vector, \(w_\theta\) is a smooth coordinate warp, and \(k_0\) is a base kernel. Let \(\mu_{t-1,\theta}(x)\) and \(\sigma_{t-1,\theta}(x)\) be the GP posterior mean and standard deviation after \(t-1\) trials under kernel \(k_\theta\). Lemma 9 states that for a fixed kernel, if \(\beta_t\ge 2B^2+300\gamma_t(k,X)\log^3(t/\delta)\), then with probability at least \(1-\delta\), \(|f(x)-\mu_{t-1}(x)|\le\sqrt{\beta_t}\sigma_{t-1}(x)\) for every \(t,x\). Here \(B\) bounds the RKHS norm of the objective, \(\gamma_t\) is maximum information gain, and \(X\) is the search domain. Applying this to all \(N=|\Theta|\) kernels and using a union bound replaces \(\delta\) by \(\delta/N\): \(\beta_t\ge 2B^2+300\gamma_t(k_\theta,X)\log^3(tN/\delta)\). Lemma 7 gives \(\sum_{t=1}^T\sigma_{t-1,\theta}^2(x_t)\le C_{\mathrm{info}}\gamma_T(k_\theta,X)\), with \(C_{\mathrm{info}}=2/\log(1+\sigma^{-2})\), observation noise variance \(\sigma^2\), and normalized kernel \(k(x,x)\le1\). Lemma 8 gives \(\gamma_T(k_\theta,X)=\gamma_T(k_0,w_\theta(X))\le\gamma_T(k_0,\mathcal X)\), so warping does not worsen the base information-gain upper bound. Use \(a_{t,\theta}(x)=\mu_{t-1,\theta}(x)+\sqrt{\beta_t}\sigma_{t-1,\theta}(x)\) and select the maximizing pair over \(\theta,x\).

## Key formulas

- $$k_\theta(x,x')=k_0\!\left(w_\theta(x),w_\theta(x')\right),\qquad a_{t,\theta}(x)=\mu_{t-1,\theta}(x)+\sqrt{\beta_t}\,\sigma_{t-1,\theta}(x)$$
- $$\beta_t\ge 2B^2+300\gamma_t(k_\theta,X)\log^3\!\left(\frac{tN}{\delta}\right)\quad\Longrightarrow\quad |f(x)-\mu_{t-1,\theta}(x)|\le\sqrt{\beta_t}\,\sigma_{t-1,\theta}(x)$$
- $$\sum_{t=1}^{T}\sigma_{t-1,\theta}^{2}(x_t)\le C_{\mathrm{info}}\gamma_T(k_\theta,X),\qquad C_{\mathrm{info}}=\frac{2}{\log(1+\sigma^{-2})}$$
- $$\gamma_T(k_\theta,X)=\gamma_T(k_0,w_\theta(X))\le\gamma_T(k_0,\mathcal X)$$

## Implementation notes

Integrate this as the outer-loop scheduler for expensive neural-network training or fine-tuning, not inside backpropagation. Represent each trial by a bounded vector \(x\in X\), for example log-learning-rate, log-weight-decay, batch-size ratio, dropout, depth, width multiplier, and optimizer choice encoded numerically. Construct a small finite library: identity; positive-log warp \(\log(x+\epsilon)\) for scale parameters; sigmoid warp for bounded parameters; signed-log warp around a reference; and one or two localized power warps. Normalize every warped coordinate before applying an RBF or Matérn base kernel. At round \(t\), collect all previous trial pairs \((x_i,y_i)\), fit or update one GP posterior per warp using the same data, estimate each \(\gamma_t(k_\theta,X)\) conservatively with a fixed common upper bound or use the largest observed posterior information gain, and set \(\beta_t=2B^2+300\bar\gamma_t\log^3(tN/\delta)\). Optimize \(a_{t,\theta}(x)=\mu_{t-1,\theta}(x)+\sqrt{\beta_t}\sigma_{t-1,\theta}(x)\) over both the library index and hyperparameters using multistart L-BFGS or dense candidate evaluation. Launch the selected neural-network trial, record validation loss or negative reward, and append it to every GP. The paper's mathematical contribution is the finite-library union-bound confidence rule and the variance/information-gain control; estimate \(B\), noise \(\sigma\), and acquisition maxima empirically. First test on CIFAR-10 ResNet-18 tuning with 50--100 trials, comparing raw-coordinate GP-UCB, manually log-scaled GP-UCB, random search, and the warp bank. Success is lower best validation error at equal completed training runs, faster improvement in the first 20 trials, and fewer failures on objectives with a narrow optimum in log-learning-rate or weight-decay coordinates.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.