Hermite-critical residual initialization
Implementation & benchmark of arXiv:2609.03589 — Correlated initialization of deep residual networks
Source paper: Correlated initialization of deep residual networks arXiv:2609.03589 ⓘ · analyzed Sep 4, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace independent residual-block parameters by a stationary correlated sequence and set the residual multiplier according to the sequence's long-memory exponent and Hermite rank. This preserves a nontrivial O(1) input-output transformation as depth grows, while avoiding activation explosion or identity collapse caused by inappropriate residual scaling.
Formulas
Mathematical statement
The network recursion is h_{l+1}=h_l+\lambda_L\sigma(w_l,h_l)v_l, where h_l is the hidden state, \sigma is the residual feature map, w_l and v_l are layer parameters, L is depth, and \lambda_L is the residual scale. Let g_l be a stationary Gaussian sequence with correlation \rho(k)=\mathbb{E}[g_0g_k]\sim c k^{-\alpha}, with c>0 and 0<\alpha<1. If a scalar parameter feature is w_l=\phi(g_l), and \phi has Hermite rank q, meaning q is the smallest integer with nonzero q-th Hermite coefficient, then the critical exponent is H=1-\alpha q/2 and the critical scaling is \lambda_L=L^{-H}. At this scaling, the cumulative random residual increments have a nontrivial limiting driver: fractional Brownian motion for q=1 and a higher-order Hermite process for q>1. The engineering reason is that the partial sum \sum_{l=0}^{L-1}(w_l-\mathbb{E}w_l) has typical size L^H, so multiplication by L^{-H} keeps the accumulated perturbation at order one. For iid finite-variance initialization, H=1/2 is the Brownian reference case.
Implementation notes
Integrate this at initialization of a preactivation ResNet or residual MLP, not inside the optimizer. For every parameter tensor intended to vary across depth, generate one correlated latent sequence for each tensor entry or, more cheaply, for low-rank channel groups. A practical MVP uses q=1 and fractional Gaussian noise. Generate a length-L Gaussian vector g with the desired long-memory covariance using FFT circulant embedding or a Cholesky factor for small L; standardize it to zero mean and unit variance. Set W_l=sigma_W g_l and optionally V_l=sigma_V times independent Gaussian noise. Use lambda=L^{-H}, with H=1-alpha/2. For q>1, transform g_l with the probabilists' Hermite polynomial He_q(g_l), normalize the transformed sequence, set H=1-alpha q/2, and again use lambda=L^{-H}. Pseudocode: sample g[0:L]; normalize g; z=Hermite_q(g); normalize z; for each block l set W_l=sigma_W z[l] and residual[l](x)=x+lambda*F_l(x,W_l). Estimate achieved correlation decay from the generated sequence and estimate H from block-sum variances. First experiment: compare 32-, 64-, and 128-block CIFAR-10 preactivation ResNets against iid He initialization, 1/sqrt(L) scaling, and 1/L scaling. Track activation variance, gradient norms, loss at equal updates, and test accuracy. Success means depth-invariant statistics and faster loss descent or higher accuracy without batch normalization.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.