{
 "artifacts": null,
 "category": "optimization",
 "description": "Replace a single-coordinate GP surrogate used for expensive neural-network hyperparameter trials with a finite bank of input-warped kernels. Each trial may select whichever warp currently gives the most promising upper-confidence acquisition, including after observing all previous results, while confidence parameters are enlarged to account for the finite library. This should recover the benefits of log-scaled or localized geometries without committing to one hand-designed parameterization.",
 "formulas_latex": [
  "$$k_\\theta(x,x')=k_0\\!\\left(w_\\theta(x),w_\\theta(x')\\right),\\qquad a_{t,\\theta}(x)=\\mu_{t-1,\\theta}(x)+\\sqrt{\\beta_t}\\,\\sigma_{t-1,\\theta}(x)$$",
  "$$\\beta_t\\ge 2B^2+300\\gamma_t(k_\\theta,X)\\log^3\\!\\left(\\frac{tN}{\\delta}\\right)\\quad\\Longrightarrow\\quad |f(x)-\\mu_{t-1,\\theta}(x)|\\le\\sqrt{\\beta_t}\\,\\sigma_{t-1,\\theta}(x)$$",
  "$$\\sum_{t=1}^{T}\\sigma_{t-1,\\theta}^{2}(x_t)\\le C_{\\mathrm{info}}\\gamma_T(k_\\theta,X),\\qquad C_{\\mathrm{info}}=\\frac{2}{\\log(1+\\sigma^{-2})}$$",
  "$$\\gamma_T(k_\\theta,X)=\\gamma_T(k_0,w_\\theta(X))\\le\\gamma_T(k_0,\\mathcal X)$$"
 ],
 "id": 3118,
 "implementation": "Integrate this as the outer-loop scheduler for expensive neural-network training or fine-tuning, not inside backpropagation. Represent each trial by a bounded vector \\(x\\in X\\), for example log-learning-rate, log-weight-decay, batch-size ratio, dropout, depth, width multiplier, and optimizer choice encoded numerically. Construct a small finite library: identity; positive-log warp \\(\\log(x+\\epsilon)\\) for scale parameters; sigmoid warp for bounded parameters; signed-log warp around a reference; and one or two localized power warps. Normalize every warped coordinate before applying an RBF or Matérn base kernel. At round \\(t\\), collect all previous trial pairs \\((x_i,y_i)\\), fit or update one GP posterior per warp using the same data, estimate each \\(\\gamma_t(k_\\theta,X)\\) conservatively with a fixed common upper bound or use the largest observed posterior information gain, and set \\(\\beta_t=2B^2+300\\bar\\gamma_t\\log^3(tN/\\delta)\\). Optimize \\(a_{t,\\theta}(x)=\\mu_{t-1,\\theta}(x)+\\sqrt{\\beta_t}\\sigma_{t-1,\\theta}(x)\\) over both the library index and hyperparameters using multistart L-BFGS or dense candidate evaluation. Launch the selected neural-network trial, record validation loss or negative reward, and append it to every GP. The paper's mathematical contribution is the finite-library union-bound confidence rule and the variance/information-gain control; estimate \\(B\\), noise \\(\\sigma\\), and acquisition maxima empirically. First test on CIFAR-10 ResNet-18 tuning with 50--100 trials, comparing raw-coordinate GP-UCB, manually log-scaled GP-UCB, random search, and the warp bank. Success is lower best validation error at equal completed training runs, faster improvement in the first 20 trials, and fewer failures on objectives with a narrow optimum in log-learning-rate or weight-decay coordinates.",
 "math_summary": "For a finite warp library indexed by \\(\\theta\\in\\Theta\\), define \\(k_\\theta(x,x')=k_0(w_\\theta(x),w_\\theta(x'))\\), where \\(x\\) is a neural hyperparameter vector, \\(w_\\theta\\) is a smooth coordinate warp, and \\(k_0\\) is a base kernel. Let \\(\\mu_{t-1,\\theta}(x)\\) and \\(\\sigma_{t-1,\\theta}(x)\\) be the GP posterior mean and standard deviation after \\(t-1\\) trials under kernel \\(k_\\theta\\). Lemma 9 states that for a fixed kernel, if \\(\\beta_t\\ge 2B^2+300\\gamma_t(k,X)\\log^3(t/\\delta)\\), then with probability at least \\(1-\\delta\\), \\(|f(x)-\\mu_{t-1}(x)|\\le\\sqrt{\\beta_t}\\sigma_{t-1}(x)\\) for every \\(t,x\\). Here \\(B\\) bounds the RKHS norm of the objective, \\(\\gamma_t\\) is maximum information gain, and \\(X\\) is the search domain. Applying this to all \\(N=|\\Theta|\\) kernels and using a union bound replaces \\(\\delta\\) by \\(\\delta/N\\): \\(\\beta_t\\ge 2B^2+300\\gamma_t(k_\\theta,X)\\log^3(tN/\\delta)\\). Lemma 7 gives \\(\\sum_{t=1}^T\\sigma_{t-1,\\theta}^2(x_t)\\le C_{\\mathrm{info}}\\gamma_T(k_\\theta,X)\\), with \\(C_{\\mathrm{info}}=2/\\log(1+\\sigma^{-2})\\), observation noise variance \\(\\sigma^2\\), and normalized kernel \\(k(x,x)\\le1\\). Lemma 8 gives \\(\\gamma_T(k_\\theta,X)=\\gamma_T(k_0,w_\\theta(X))\\le\\gamma_T(k_0,\\mathcal X)\\), so warping does not worsen the base information-gain upper bound. Use \\(a_{t,\\theta}(x)=\\mu_{t-1,\\theta}(x)+\\sqrt{\\beta_t}\\sigma_{t-1,\\theta}(x)\\) and select the maximizing pair over \\(\\theta,x\\).",
 "math_tags": [
  "probability",
  "information-theory",
  "statistics",
  "approximation-theory"
 ],
 "ml_areas": [
  "fine-tuning",
  "optimizer",
  "scheduler"
 ],
 "paper": {
  "arxiv_id": "2609.02993",
  "arxiv_url": "https://arxiv.org/abs/2609.02993",
  "summary_what_math_gives_to_ml": "The paper provides a constructive way to adapt the geometry of expensive black-box search without sacrificing high-probability guarantees: maintain a finite library of warped kernels and permit arbitrary history-dependent kernel selection. The key transferable asset is that fixed-kernel confidence bounds can be lifted to the entire library by a union bound, while information gain under a warp is no larger than that of the base kernel on the warped domain. This is directly useful for neural-network hyperparameter and fine-tuning searches, where learning-rate scales, weight-decay scales, batch sizes, and architectural ratios often have radically different geometries.",
  "title": "No-Regret Bayesian Optimization with Finite-Library Input-Warped Kernels",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 6,
  "usefulness": 6
 },
 "solves": [
  "sample-efficiency",
  "accuracy",
  "stability"
 ],
 "title": "Finite-Warp UCB for Neural Hyperparameter Search",
 "url": "https://synthcore.org/idea/3118/finite-warp-ucb-for-neural-hyperparameter-search",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}