✓✓ Beats tuned baseline 2026

Jacobian-aligned infill for black-box neural tuning

Implementation & benchmark of arXiv:2608.22862 — JANUS: Online Jacobian-Aligned Infill for Black-Box Optimization

Usefulness6/10

Difficulty5/10

Novelty7/10

Optimization Sample efficiency Speedup Accuracy Optimizer Fine tuning Quantization Scheduler Optimization Linear algebra Numerical analysis

Source paper: JANUS: Online Jacobian-Aligned Infill for Black-Box Optimization arXiv:2608.22862 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add a geometry-guided infill operator to a population optimizer used for black-box neural-network tuning. Fit a local Jacobian from recent parameter perturbations and validation-residual vectors, generate a damped Gauss-Newton candidate for exploitation, and sample exploratory candidates in the same Jacobian-derived metric. The host optimizer retains selection, population survival, covariance adaptation, and its total evaluation budget; only a configurable fraction of new candidates is replaced by geometry-guided infill.

Formulas

$$T=(J^{\top}J+\lambda I)^{-1}$$

$$\hat J=\arg\min_J\sum_i w_i\left\|r_i-r_0-J(z_i-z_0)\right\|_2^2+\eta\|J\|_F^2$$

$$\delta_{\mathrm{GN}}=-(\hat J^{\top}\hat J+\lambda I)^{-1}\hat J^{\top}r_0$$

$$\delta_{\mathrm{explore}}=\rho\,\frac{T^{1/2}u}{\|T^{1/2}u\|_2},\qquad u\sim\mathcal N(0,I),\quad T=(\hat J^{\top}\hat J+\lambda I)^{-1}$$

Mathematical statement

The paper identifies the inverse local Gauss-Newton metric T=(J^T J+lambda I)^(-1), where J is the Jacobian of the locally modeled residual vector with respect to the search variables and lambda>0 is damping. Let z in R^p be the black-box tuning vector, r(z) in R^m a residual vector formed from validation losses on m fixed shards, and (z_i,r_i) recent evaluations. Estimate J by weighted ridge regression around the incumbent z_0: minimize the weighted squared linearization error plus eta times the squared Frobenius norm, where w_i are locality weights and eta>0 stabilizes underdetermined fits. The resulting Gauss-Newton residual-model step is delta_GN=-(J_hat^T J_hat+lambda I)^(-1)J_hat^T r_0. Exploration uses the same metric, with a random direction transformed by T^(1/2), so directions of low local residual sensitivity receive larger steps. Proposition 1 states that a covariance basis fixed independently of J cannot in general align its exploration metric with this inverse-Hessian approximation; the transferable property is therefore online, trace-based metric adaptation.

Implementation notes

Integrate this at the candidate-generation stage of CMA-ES or differential evolution, not inside backpropagation. Let the search vector z contain 10–100 black-box neural tuning variables such as LoRA adapter scales, per-layer learning-rate multipliers, clipping thresholds, quantization scales, dropout rates, or compact architecture choices. Evaluate each candidate on the same m validation shards and store the residual vector r_i, for example normalized shard losses relative to a fixed baseline. Maintain a FIFO trace of K evaluations with K at least p+m. At each generation, choose the incumbent z_0 and residual r_0, compute Delta z_i=z_i-z_0, and fit the ridge Jacobian J_hat using weights w_i=exp(-||Delta z_i||^2/h^2). Form A=J_hat^T J_hat+lambda I and solve A delta=-J_hat^T r_0 using Cholesky or conjugate gradients; do not explicitly invert A. Clip delta to trust radius rho, project it to legal parameter bounds, and insert it as one exploitation candidate. For exploration, sample u from a standard Gaussian and obtain T^(1/2)u by solving with the Cholesky factor of A, then normalize and scale by rho. Fill remaining slots with the unchanged host sampler and let the host select survivors. Increase lambda or shrink rho when predicted and observed residual decreases disagree; disable infill when the Jacobian condition estimate is poor. Estimate h, lambda, rho, and the infill fraction empirically, while computing J_hat, A, the Gauss-Newton step, and metric-shaped samples from the stated formulas. First test on CIFAR-10 using a frozen small ResNet and black-box tuning of 20 LoRA, quantization, and schedule variables, with CMA-ES as baseline. Use equal validation-evaluation budgets and measure best accuracy versus evaluations, wall-clock overhead, prediction error, and ablations removing either exploitation or exploration. Success is faster accuracy improvement at equal evaluations, at least 10% fewer evaluations to reach a fixed accuracy, and no severe failures on ill-conditioned local fits.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built Jacobian fitting, damped Gauss–Newton exploitation, and inverse-metric exploration infill in experiment.py. The exact linear check recovered the Jacobian with relative error 5.3e-11, reduced residual norm to 30.8% of its initial value, and confirmed inverse-metric eigenvalue alignment. In the paired 8-seed toy benchmark, guided infill reached mean loss 2.99e-32 versus 2.32 for the control after 500 evaluations, with much faster improvement; this is promising toy evidence, not neural-tuning validation.

Agent confidence: 8/10

Baseline: Mean best loss at evaluations [100, 200, 300, 400, 500]: [19.893, 5.886, 3.199, 2.664, 2.319]; final standard deviation 0.658.
Idea: Mean best loss at evaluations [100, 200, 300, 400, 500]: [3.899, 5.72e-9, 2.21e-18, 1.25e-27, 2.99e-32]; final standard deviation 0.0. Jacobian relative error 5.27e-11; Gauss–Newton residual ratio 0.308; metric alignment check true.

Limitations: The experiment uses an 8-dimensional synthetic nonlinear residual function, not CIFAR-10, a neural network, LoRA, quantization variables, CMA-ES, or differential evolution. The control is isotropic random incumbent perturbation rather than a full CMA-ES implementation. Wall-clock overhead, prediction-error adaptation, ablations, and broad ill-conditioning stress tests were not evaluated.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.22862) Sep 3, 2026

Beats the tuned baseline by 24.3% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 10.8468
Idea mean: 8.2086
Effect (Δ): -2.6382 (−24.3%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±6.4%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 5 configs

Limitations:

The experiment used the tabular track only, with a small 800-sample training subset, 300 test samples, mlp_tiny, and 12 epochs. The baseline did not receive an explicitly matched number of pilot/candidate evaluations, so the observed advantage may include extra evaluation budget. No vision, sequence, dynamics, wall-clock, ablation, or broader ill-conditioning study was run. The auxiliary exact-linear check should be corrected before treating the mathematical verification as valid.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 3, 2026 latest Worked ✓

Agent confidence: 7/10

Implemented Jacobian fitting, damped Gauss–Newton exploitation, inverse-metric exploration, and a tabular neural benchmark using the read-only bench harness. The paired 8-seed comparison reports a significant improvement in test MSE, with baseline 10.8468 versus idea 8.2086, delta -2.6382 and permutation p=0.0081. This is a reported win, but confidence is reduced because the idea evaluates pilot/candidate models beyond the baseline's matched evaluation count, and the auxiliary exact-linear math check in the script was incorrect and failed.

Baseline: mlp_tiny on tabular/Friedman#1; 12 epochs; lr sweep [0.001, 0.002, 0.003, 0.005, 0.008]; full 8-seed mean test MSE 10.8468 ± 0.9461.
Idea: Jacobian-guided learning-rate infill around baseline lr 0.008; full 8-seed mean test MSE 8.2086 ± 0.5743; paired delta -2.6382; permutation p=0.0081; idea won on all 8 seeds.
Math check: The generated auxiliary exact-linear check reported relative error 0.8607, so that check did not verify accurate Jacobian recovery.
Mechanism signature: Pilot validation mean 7.5602; Gauss–Newton predicted validation mean 8.9957; observed mean 8.7001; absolute prediction error 0.5079; condition estimate 1.0; reported confirmed=true.

Benchmark result