Correlated initialization of deep residual networks

What the math gives to ML

The paper identifies a concrete depth-dependent initialization law for residual networks with long-range correlated layer weights: if layer correlations decay as k^{-\alpha} and the feature map has Hermite rank q, the accumulated residual noise scales like L^H with H=1-\alpha q/2, so the nontrivial regime uses \lambda_L\asymp L^{-H}. This gives a principled continuum between Brownian-like independent initialization and smoother, strongly correlated depth dynamics, rather than choosing the usual 1/L or 1/\sqrt{L} scaling heuristically. The most transferable asset is a generator for correlated parameters together with a scaling rule that preserves an O(1) feature evolution as depth changes. A practical first test is to compare ordinary iid residual initialization against fractional-Gaussian or Hermite-transformed layer noise at equal depth, parameter count, and marginal activation variance.