Singular perturbations and hierarchical learning in two-layer neural networks

What the math gives to ML

The paper analyzes hierarchical learning in a two-layer network through singularly perturbed gradient flows. Its transferable asset is the separation between fast output-weight relaxation and slow hidden-feature evolution: fast variables approach an integral-constraint manifold while slow variables evolve using the already-learned components. This suggests a principled layerwise optimizer schedule in which output coefficients first fit what the current features can represent, followed by progressively faster feature learning after residual plateaus. The approach is especially testable on synthetic single-index regression, where linear and quadratic target components provide direct diagnostics of the predicted learning order.