✗ Failed on benchmark 2026

Hermite-critical residual initialization

Implementation & benchmark of arXiv:2609.03589 — Correlated initialization of deep residual networks

Usefulness7/10

Difficulty6/10

Novelty8/10

Dynamics Stability Accuracy Scalability Resnet Initialization Training dynamics Stochastic processes Probability Dynamical systems

Source paper: Correlated initialization of deep residual networks arXiv:2609.03589 ⓘ · analyzed Sep 4, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace independent residual-block parameters by a stationary correlated sequence and set the residual multiplier according to the sequence's long-memory exponent and Hermite rank. This preserves a nontrivial O(1) input-output transformation as depth grows, while avoiding activation explosion or identity collapse caused by inappropriate residual scaling.

Formulas

$$h_{l+1}=h_l+\lambda_L\sigma(w_l,h_l)v_l,\qquad l=0,\dots,L-1.$$

$$\rho(k)=\operatorname{Corr}(g_0,g_k)\sim c k^{-\alpha},\qquad H=1-\frac{\alpha q}{2}.$$

$$\gamma=H=1-\frac{\alpha q}{2},\qquad \lambda_L=L^{-\gamma}=L^{-H}.$$

$$\sum_{l=0}^{L-1}(w_l-\mathbb{E}w_l)=O_{\mathrm{typ}}(L^H),\qquad \lambda_L\sum_{l=0}^{L-1}(w_l-\mathbb{E}w_l)=O_{\mathrm{typ}}(1).$$

Mathematical statement

The network recursion is h_{l+1}=h_l+\lambda_L\sigma(w_l,h_l)v_l, where h_l is the hidden state, \sigma is the residual feature map, w_l and v_l are layer parameters, L is depth, and \lambda_L is the residual scale. Let g_l be a stationary Gaussian sequence with correlation \rho(k)=\mathbb{E}[g_0g_k]\sim c k^{-\alpha}, with c>0 and 0<\alpha<1. If a scalar parameter feature is w_l=\phi(g_l), and \phi has Hermite rank q, meaning q is the smallest integer with nonzero q-th Hermite coefficient, then the critical exponent is H=1-\alpha q/2 and the critical scaling is \lambda_L=L^{-H}. At this scaling, the cumulative random residual increments have a nontrivial limiting driver: fractional Brownian motion for q=1 and a higher-order Hermite process for q>1. The engineering reason is that the partial sum \sum_{l=0}^{L-1}(w_l-\mathbb{E}w_l) has typical size L^H, so multiplication by L^{-H} keeps the accumulated perturbation at order one. For iid finite-variance initialization, H=1/2 is the Brownian reference case.

Implementation notes

Integrate this at initialization of a preactivation ResNet or residual MLP, not inside the optimizer. For every parameter tensor intended to vary across depth, generate one correlated latent sequence for each tensor entry or, more cheaply, for low-rank channel groups. A practical MVP uses q=1 and fractional Gaussian noise. Generate a length-L Gaussian vector g with the desired long-memory covariance using FFT circulant embedding or a Cholesky factor for small L; standardize it to zero mean and unit variance. Set W_l=sigma_W g_l and optionally V_l=sigma_V times independent Gaussian noise. Use lambda=L^{-H}, with H=1-alpha/2. For q>1, transform g_l with the probabilists' Hermite polynomial He_q(g_l), normalize the transformed sequence, set H=1-alpha q/2, and again use lambda=L^{-H}. Pseudocode: sample g[0:L]; normalize g; z=Hermite_q(g); normalize z; for each block l set W_l=sigma_W z[l] and residual[l](x)=x+lambda*F_l(x,W_l). Estimate achieved correlation decay from the generated sequence and estimate H from block-sum variances. First experiment: compare 32-, 64-, and 128-block CIFAR-10 preactivation ResNets against iid He initialization, 1/sqrt(L) scaling, and 1/L scaling. Track activation variance, gradient norms, loss at equal updates, and test accuracy. Success means depth-invariant statistics and faster loss descent or higher accuracy without batch normalization.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a reproducible Hermite-critical residual MVP with correlated Gaussian latent sequences, Hermite-rank transforms, exact finite-length variance checks, and a toy residual recursion. For alpha=0.4, exact finite-n fitted H values were 0.828 for q=1 (theory 0.8) and 0.687 for q=2 (theory 0.6); empirical estimates were 0.858 and 0.730. Critical correlated scaling stayed bounded across depths 32–256, while correlated L^-1/2 scaling grew strongly, so the claimed stability signal was observed, although no optimization or accuracy win was established.

Agent confidence: 8/10

Baseline: Correlated L^-1/2 control final variances: 6.92, 14.76, 16.10, 29.34 at depths 32, 64, 128, 256; iid L^-1/2: 1.28, 1.31, 1.53, 1.42.
Idea: Critical correlated L^-H with alpha=0.4 and H=0.8 final variances: 2.27, 1.94, 2.16, 2.11 at depths 32, 64, 128, 256. Exact fitted H values: 0.828 for q=1 and 0.687 for q=2 versus theoretical 0.8 and 0.6.

Limitations: Only a small NumPy toy recursion was tested; no CIFAR-10, ResNet, optimizer training, loss descent, gradient norms, accuracy, wall-clock comparison, FFT sequence generation, or large-depth asymptotic validation was performed. The finite-depth q=2 estimate is noisy and shows noticeable finite-size deviation from its asymptotic exponent.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.03589) Sep 4, 2026

Significantly WORSE than the tuned baseline (+7.6%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: local_residual_mlp
Paired seeds: 8
Baseline mean: 6.5375
Idea mean: 7.0365
Effect (Δ): 0.499 (+7.6%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±10.6%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 6 configs

Limitations:

Only the mandatory structurally appropriate tabular track was tested; no CIFAR-10, sequence, dynamics, q>1 neural-network variant, deeper residual networks, or longer training budgets were evaluated. The local implementation uses a small depth-8 residual MLP and finite-length Cholesky correlation generation rather than FFT generation or large-depth asymptotics.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 9/10

Built and evaluated a matched residual MLP on the Friedman#1 tabular benchmark using the canonical bench training loop, an 8-seed paired comparison, and a parity-preserving baseline sweep. The mathematical scaling check passed (theory H=0.8, observed H=0.821, scaled RMS ratio=1.063), but the idea had higher test MSE than the tuned iid baseline (7.037 vs 6.537; paired delta +0.499, p=0.0081) and lost on all 8 seeds. Therefore the stability signal does not translate into a benchmark training improvement in this setup.

Baseline: Friedman#1 tabular residual MLP, iid gates, best swept config lr=0.006, exponent=0.5, 18 epochs; full 8-seed mean test MSE=6.5375, std=0.7963.
Idea: Same residual MLP/data/Adam/batch/epochs and lr union, correlated Gaussian gates with alpha=0.4 and best idea exponent=0.7; full 8-seed mean test MSE=7.0365, std=0.6551.
Math check: H theory=0.8, observed H=0.8213, scaled-sum RMS values=[1.3915,1.2675,1.4158,1.2996,1.4795], last/first=1.0632, passed=true.
Mechanism signature: Predicted adjacent gate correlation=0.7579, observed mean across trained benchmark systems=0.0362, confirmed=false.
Paired delta: idea - baseline = +0.4990 MSE; idea wins 0/8 seeds; permutation p=0.0081; verdict=idea worse (significant).

Benchmark result