Order-Parameter Mode Activation Schedule
Implementation & benchmark of arXiv:2608.13335 — Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
Source paper: Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws arXiv:2608.13335 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Use the paper's order-parameter dynamics to initialize spectral feature modes with deliberately separated activation times. This creates a controlled progressive-learning curriculum in which dominant modes become available first and weaker modes activate later, potentially reducing early gradient interference.
Formulas
Mathematical statement
The paper uses the collective state M = W W^T and states that, when the relevant data matrices share an eigenbasis, training closes on scalar eigenmodes and reduces to a Lotka-Volterra-type equation. Let m_k be the nonnegative amount of order parameter in eigenmode k, r_k its uncoupled growth rate, and K_kj the coupling or competition coefficient between modes k and j. The resulting mode equation is dot(m_k) = m_k (r_k - sum_j K_kj m_j). The multiplicative factor m_k makes zero invariant and causes small initial modes to grow slowly. When competition is initially negligible, the time to reach a target m_k,target is approximately t_k = log(m_k,target / m_k(0)) / r_k. Therefore setting m_k(0) to decrease geometrically with k produces separated onset times.
Implementation notes
Apply this to a quadratic module or to spectral channels of an MLP projection, attention value projection, or convolutional channel mixer. During a short calibration pass, collect hidden features h_b and estimate their covariance C = (1/N) sum_b h_b h_b^T. Compute the top r eigenvectors Q of C and use them as approximate shared data eigenmodes. Parameterize an initialization W = Q diag(sqrt(m_0)) R^T, where R is a small random mixing matrix and m_0 contains the desired mode energies. Set m_k(0) = epsilon_0^2 exp(-2 alpha k), as suggested by the onset-time equation. Pseudocode: C = running_covariance(features); eigenvalues,Q = eigh(C); Q = top_r(Q); m0 = eps0**2 * exp(-2*alpha*arange(r)); W = Q @ diag(sqrt(m0)) @ R.T; train normally; m = diag(Q.T @ W @ W.T @ Q). Estimate r_k from the initial slope of log(m_k + delta) over the first 20 to 100 minibatches; estimate K only if needed by fitting the observed mode derivatives to the linear form r_k - sum_j K_kj m_j. The mathematical prediction is the multiplicative mode law and logarithmic onset time; the shared eigenbasis is only approximate and must be measured. First test on a 2-layer MLP or a 2-layer Transformer on CIFAR-10 or a small language-model corpus, comparing isotropic initialization against alpha values from 0 to 2. Log loss, gradient variance, mode energies, mode participation entropy, and predicted versus observed onset times. Success is lower early gradient interference, smoother loss, or faster loss reduction at equal FLOPs without sacrificing final accuracy. A negative result is no correlation between predicted and observed onset times or slower convergence for all alpha values.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.