Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
arXiv:2608.13335
2026
Architecture
2 ideas extracted · analyzed Sep 1, 2026
What the math gives to ML
The paper identifies a universality mechanism in which permutation symmetry across interchangeable units, together with vanishing per-unit gradients at zero, forces the leading near-initialization computation to depend on weights through the Gram matrix M = WW^T. This collapses many architectures into a quadratic feature family Tr(M A(x)), where A(x) contains the architecture-specific information. The construction suggests replacing wide sums of exchangeable branches with explicit low-rank order-parameter modules. Its mode-wise dynamics also suggest a principled way to create progressive feature activation through initialization or scheduling.
Ideas from this paper
✗ Failed on benchmark
2026
Replace a wide collection of interchangeable near-zero branches with a module whose output is explicitly a quadratic form in the branch-weight Gram matrix. The module preserves the paper's leading-order behavior while making the relevant collective variable explicit and allowing low-rank parameterizations.
Useful7/10
Difficulty5/10
Novelty7/10
Unverified
2026
Use the paper's order-parameter dynamics to initialize spectral feature modes with deliberately separated activation times. This creates a controlled progressive-learning curriculum in which dominant modes become available first and weaker modes activate later, potentially reducing early gradient interference.
Useful6/10
Difficulty6/10
Novelty6/10