△ Mechanism confirmed, baseline not beaten 2026

Multilevel Neural Trace Control Variate

Implementation & benchmark of arXiv:2607.05157 — Variance reduction with probing and Multilevel Monte Carlo in Lattice QCD

Usefulness7/10

Difficulty6/10

Novelty6/10

Optimization Speedup Scalability Optimizer Regularization Transformer Training dynamics Probability Stochastic processes Numerical analysis Linear algebra Optimization

Source paper: Variance reduction with probing and Multilevel Monte Carlo in Lattice QCD arXiv:2607.05157 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Estimate an expensive fine-model trace or quadratic-form quantity using a telescoping sum over cheap-to-expensive neural approximations. Allocate many probes to cheap levels and only a few probes to the expensive level, exploiting strong correlation between adjacent levels to reduce variance at fixed compute. Candidate levels include truncated Transformer depth, reduced width, low-rank curvature, coarser graph resolution, or progressively tighter implicit-solver tolerances.

Formulas

$$\mathbb{E}[Y_L]=\mathbb{E}[Y_0]+\sum_{\ell=1}^{L}\mathbb{E}[Y_\ell-Y_{\ell-1}],\qquad \Delta_0=Y_0,\quad \Delta_\ell=Y_\ell-Y_{\ell-1}.$$

$$\widehat Y_{\mathrm{ML}}=\sum_{\ell=0}^{L}\frac{1}{N_\ell}\sum_{k=1}^{N_\ell}\Delta_\ell^{(k)},\qquad \Delta_\ell^{(k)}=\eta_{\ell,k}^{*}(A_\ell-A_{\ell-1})\eta_{\ell,k}.$$

$$\operatorname{Var}(\widehat Y_{\mathrm{ML}})=\sum_{\ell=0}^{L}\frac{V_\ell}{N_\ell},\qquad N_\ell\propto\sqrt{\frac{V_\ell}{C_\ell}},\quad V_\ell=\operatorname{Var}(\Delta_\ell).$$

Mathematical statement

Let $Y_L$ be the stochastic trace contribution from the finest approximation and $Y_0,\ldots,Y_{L-1}$ progressively cheaper approximations coupled using the same random probe. Define $\Delta_0=Y_0$ and $\Delta_\ell=Y_\ell-Y_{\ell-1}$. The multilevel identity is $\mathbb{E}[Y_L]=\mathbb{E}[Y_0]+\sum_{\ell=1}^{L}\mathbb{E}[Y_\ell-Y_{\ell-1}]$, so the estimator is unbiased when the finest level is the desired operator and all levels use the same probe distribution. If $V_\ell=\operatorname{Var}(\Delta_\ell)$ and $C_\ell$ is the cost of one coupled sample, the fixed-cost variance-minimizing allocation satisfies $N_\ell\propto\sqrt{V_\ell/C_\ell}$. Correlation between adjacent approximations makes $V_\ell$ small, while cheap coarse levels can receive many samples.

Implementation notes

(1) Integration point: add a multilevel estimate of $\operatorname{tr}(H)$, $\operatorname{tr}(F)$, or another curvature operator inside a training loop. Define levels by truncating a Transformer to depths $d_0<d_1<\cdots<d_L$, using reduced widths, or using increasingly accurate conjugate-gradient tolerances for an implicit layer. All operators must act in the same parameter space; if a coarse model has fewer coordinates, embed it with a prolongation map $P_\ell$. (2) Pseudocode: for each shared probe $\eta$, evaluate $Y_\ell=\eta^*A_\ell\eta$ at adjacent levels; set $\Delta_0=Y_0$ and $\Delta_\ell=Y_\ell-Y_{\ell-1}$; average $N_\ell$ samples per level and sum the averages. Begin with 10 pilot samples per level, measure sample variances $\widehat V_\ell$ and costs $\widehat C_\ell$, then allocate subsequent samples proportional to $\sqrt{\widehat V_\ell/\widehat C_\ell}$ under a fixed HVP or wall-clock budget. (3) The telescoping identity and unbiasedness are mathematical; variance, cost, and level allocation are estimated empirically. Reject a level hierarchy if adjacent estimates have weak correlation. (4) First experiment: use a six-layer tiny Transformer with one-, two-, four-, and six-layer levels. Estimate Hessian trace every 100 steps and compare 100 fine-level Hutchinson probes against the adaptive multilevel estimator at equal HVP cost. Measure variance, bias against a 1000-probe reference, wall-clock time, and downstream accuracy when used as a sharpness penalty. Success is at least 3x lower variance or 3x less time to reach a fixed trace-estimation error without measurable bias.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a self-contained multilevel Hutchinson trace estimator with coupled Gaussian probes, adaptive allocation proportional to sqrt(V_l/C_l), and numerical verification. The telescoping identity passed with 6.7e-5 relative Monte Carlo error; variance-decay sweeps matched the exact boundary formula and decay^2 fine-level prediction, adjacent correlations exceeded 0.998, and RMSE followed N^-1/2 scaling. At equal synthetic cost, adaptive multilevel variance was 0.227 versus 0.520 for fine-only, a 2.29x reduction; this is a real toy-setting signal, though below the proposed 3x target.

Agent confidence: 9/10

Baseline: Fine-only Hutchinson: variance=0.5200, RMSE=0.7193, bias=0.0248, 414 finest probes at budget 3200
Idea: Adaptive multilevel: variance=0.2270, RMSE=0.4756, bias=0.0238, allocation=[2212,144,77,44,27], variance reduction=2.29x versus fine-only; equal-probe multilevel variance=2.2020
Math checks: Telescoping relative error=6.68e-5; adjacent correlations=[0.99968,0.99977,0.99985,0.99991]; variance-decay and probe-scaling checks passed

Limitations: This is a synthetic diagonal-operator toy experiment, not a Transformer Hessian or GPU HVP implementation. Costs are prescribed proxies rather than measured wall-clock costs, only Gaussian Hutchinson probes were tested, and downstream training accuracy or sharpness-penalty effects were not evaluated.

How to run: python3 multilevel_trace_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.05157) Sep 2, 2026

Ahead of the baseline (−0.8%) but not statistically significant (p=0.7398, wins 3 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 15.3892
Idea mean: 15.264
Effect (Δ): -0.1251 (−0.8%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.7398 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±5.0%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 9 configs

Limitations:

Only the structurally matched built-in tabular track was tested; vision, sequence, and dynamics were not tested. The signature records a finite coupled HVP trace but does not independently estimate adjacent-level correlation numerically, so its confirmation is limited. The implementation uses parameter-prefix truncation as the multilevel hierarchy rather than Transformer depth because the selected built-in track is tabular.

How to run: python3 run_stage2.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 10/10

Built a multilevel Hutchinson Hessian-trace penalty into the shared mlp_tiny training system and compared it against standard Adam training with matched learning-rate and weight-decay sweeps. On the tabular Friedman#1 regression benchmark, the best idea setting gave MSE 15.2640 versus baseline 15.3892 (delta -0.1251), but p=0.7398, so there is no significant win. The trained-model signature was finite and marked confirmed, but this does not override the task-metric result.

Baseline: tabular/mlp_tiny, tuned lr=0.006 and weight_decay=0.0001; 8-seed test MSE mean=15.38918, std=2.10680
Idea: trace_lambda=1e-05 with lr=0.006 and weight_decay=0.0001; 8-seed test MSE mean=15.26404, std=2.16644; delta=-0.12514; 3/8 paired wins; permutation p=0.7398

Benchmark result