Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

What the math gives to ML

The paper's transferable asset is a recursive function-space norm that measures the ℓ1 mass of feature combinations across depth, together with the normalized activation parameterization \(\sigma_s(t)=\sigma(st)/s\), which separates feature shape from output scale for non-homogeneous activations. This suggests replacing or augmenting parameter weight decay with a variation-style path regularizer that directly controls the function represented by each layer. The paper's depth-saturation and low-frequency conclusions motivate a falsifiable hypothesis: once this norm is controlled, adding depth to ReLU MLPs should provide little extra one-dimensional functional diversity, while GELU/SiLU may retain useful depth-dependent shape changes. The most practical first transfer is a recursive variation penalty implemented on hidden-layer coefficients, with ablations against ordinary L2 weight decay and an explicit measurement of directional high-frequency behavior.