Singularly Perturbed Hierarchical Training

Integrate the method into the optimizer step of a width-m two-layer MLP, modifying the learning rates of the output coefficients and hidden weights. Let eta_a be the base output learning rate and initialize the hidden rate as eta_u = epsilon times eta_a, with epsilon tested at 1e-1, 1e-2, and 1e-3. On every minibatch, compute the loss, backpropagate once, update output parameters using a_a <- a_a - eta_a g_a, and update hidden parameters using theta_u <- theta_u - eta_u g_u. If the experiment explicitly stores each hidden direction as a unit vector u_i, replace its gradient by (I - u_i u_i^T) g_u_i and renormalize after the update; for ordinary unconstrained MLP weights, omit this projection. Maintain an exponential moving average Lbar_t = beta Lbar_{t-1} + (1-beta)L_t. Every K steps, calculate the relative plateau measure r_t = abs(Lbar_t - Lbar_{t-K}) / (abs(Lbar_{t-K}) + 1e-8). If r_t remains below a threshold tau for q consecutive checks, multiply epsilon by 2, up to a maximum of 1, thereby allowing the hidden layer to enter the next learning stage. The paper determines the fast/slow structure and constraint-manifold interpretation; epsilon, beta, K, tau, q, and the maximum rate are empirical choices. First test a width-256 ReLU network on Gaussian single-index regression with targets y = c1 z + c2(z^2 - 1), z = <u_*,x>, using 10,000 training and 10,000 test samples. Compare equal learning rates, AdamW, and fixed epsilon values against the adaptive schedule at equal optimizer steps and FLOPs. Measure total test MSE, separate errors from linear and quadratic target components, cosine alignment between learned hidden directions and u_*, and gradient norms. Success means faster initial loss descent, a clear plateau followed by improved quadratic-component learning, and no loss spike when epsilon increases.