# Order-Parameter Mode Activation Schedule

- ID: 2236
- Canonical URL: https://synthcore.org/idea/2236/order-parameter-mode-activation-schedule
- API JSON: https://synthcore.org/api/idea/2236.json
- API Markdown: https://synthcore.org/api/idea/2236.md
- Verification status: unverified
- Source: [arXiv:2608.13335](https://arxiv.org/abs/2608.13335)
- Category: training
- Solves: stability, accuracy, generalization
- ML areas: training-dynamics, initialization, optimizer, mlp, attention
- Math tags: dynamical-systems, linear-algebra, symmetry
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Use the paper's order-parameter dynamics to initialize spectral feature modes with deliberately separated activation times. This creates a controlled progressive-learning curriculum in which dominant modes become available first and weaker modes activate later, potentially reducing early gradient interference.

## Mathematical statement

The paper uses the collective state M = W W^T and states that, when the relevant data matrices share an eigenbasis, training closes on scalar eigenmodes and reduces to a Lotka-Volterra-type equation. Let m_k be the nonnegative amount of order parameter in eigenmode k, r_k its uncoupled growth rate, and K_kj the coupling or competition coefficient between modes k and j. The resulting mode equation is dot(m_k) = m_k (r_k - sum_j K_kj m_j). The multiplicative factor m_k makes zero invariant and causes small initial modes to grow slowly. When competition is initially negligible, the time to reach a target m_k,target is approximately t_k = log(m_k,target / m_k(0)) / r_k. Therefore setting m_k(0) to decrease geometrically with k produces separated onset times.

## Key formulas

- $$M=WW^{\top}$$
- $$\dot m_k=m_k\left(r_k-\sum_jK_{kj}m_j\right)$$
- $$t_k\approx \frac{1}{r_k}\log\!\left(\frac{m_{k,\mathrm{target}}}{m_k(0)}\right)\quad\text{when }\sum_jK_{kj}m_j\ll r_k$$
- $$m_k(0)=\varepsilon_k^2,\qquad \varepsilon_k=\varepsilon_0\exp(-\alpha k)$$

## Implementation notes

Apply this to a quadratic module or to spectral channels of an MLP projection, attention value projection, or convolutional channel mixer. During a short calibration pass, collect hidden features h_b and estimate their covariance C = (1/N) sum_b h_b h_b^T. Compute the top r eigenvectors Q of C and use them as approximate shared data eigenmodes. Parameterize an initialization W = Q diag(sqrt(m_0)) R^T, where R is a small random mixing matrix and m_0 contains the desired mode energies. Set m_k(0) = epsilon_0^2 exp(-2 alpha k), as suggested by the onset-time equation. Pseudocode: `C = running_covariance(features); eigenvalues,Q = eigh(C); Q = top_r(Q); m0 = eps0**2 * exp(-2*alpha*arange(r)); W = Q @ diag(sqrt(m0)) @ R.T; train normally; m = diag(Q.T @ W @ W.T @ Q)`. Estimate r_k from the initial slope of log(m_k + delta) over the first 20 to 100 minibatches; estimate K only if needed by fitting the observed mode derivatives to the linear form r_k - sum_j K_kj m_j. The mathematical prediction is the multiplicative mode law and logarithmic onset time; the shared eigenbasis is only approximate and must be measured. First test on a 2-layer MLP or a 2-layer Transformer on CIFAR-10 or a small language-model corpus, comparing isotropic initialization against alpha values from 0 to 2. Log loss, gradient variance, mode energies, mode participation entropy, and predicted versus observed onset times. Success is lower early gradient interference, smoother loss, or faster loss reduction at equal FLOPs without sacrificing final accuracy. A negative result is no correlation between predicted and observed onset times or slower convergence for all alpha values.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.