Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks
arXiv:2608.06597
2026
Dynamics
2 ideas extracted · analyzed Aug 31, 2026
What the math gives to ML
The paper provides a tractable mechanism by which increasing L2 regularization causes a cascade of feature-detection phase transitions in deep linear networks. Under whitened inputs and approximately balanced, singular-vector-aligned layers, each data singular mode has a computable activation threshold determined by its target strength, depth, and regularization coefficient; the learned rank therefore changes sharply as beta varies. The most transferable construction is a regularization curriculum or rank controller that predicts these thresholds from the empirical input-output spectrum and tests whether observed feature norms and Hessian curvature exhibit the predicted transitions.
Ideas from this paper
Unverified
2026
Replace fixed weight decay with a spectrum-aware schedule that intentionally crosses predicted activation thresholds one at a time. The curriculum should first learn strong, well-conditioned input-output modes and only later lower regularization enough to activate weak modes, producing controlled rank growth instead of simultaneous fitting of noisy directions.
Useful8/10
Difficulty5/10
Novelty7/10
✗ Failed on benchmark
2026
Use the paper's explicit Hessian dependence on learned singular values to detect when a feature mode approaches a curvature transition, then adapt weight decay or learning rate before the mode destabilizes. This turns regularization from a static hyperparameter into feedback control based on mode-wise curvature and feature amplitude.
Useful7/10
Difficulty5/10
Novelty6/10