Robust Parameter-Update Envelope

Integrate this as a wrapper around AdamW, SGD, or a momentum optimizer. First choose inexpensive constraints: (i) maximum per-layer activation RMS, $g_1=\mathrm{RMS}(h_l)-b_l$; (ii) gradient norm, $g_2=\|\nabla_\theta L\|_2-G$; and (iii) optionally a power-iteration estimate of the largest layer spectral norm, $g_3=\hat\sigma_l-s_l$. Compute the usual optimizer direction $d_t$, then test $K=8$ interpolation points $\theta_t+(k/K)\eta_0d_t$ for a proposed base learning rate $\eta_0$. Evaluate a small fixed minibatch at each point, compute all $g_j$, and accept the step only if their maximum is nonpositive. If a violation occurs, halve the step size and rescan; after acceptance, optionally refine between the last feasible and first infeasible grid point with two bisection evaluations. Pseudocode: d=optimizer_direction(theta); eta=eta0; while max_constraint(theta+s*eta*d for s in grid)>0: eta*=0.5; theta += eta*d. The source-paper mechanism supplies robust feasibility over an entire envelope; in the neural-network experiment, constraints and path evaluations are estimated empirically rather than analytically certified. Start with a 2-layer MLP on MNIST or CIFAR-10, comparing AdamW with endpoint-only clipping against robust-envelope AdamW at matched initial learning rate. Log maximum interior violation, accepted step sizes, loss spikes, and gradient norms. The quantitative prediction is that endpoint-only training will show nonzero interior violations on some steps whose endpoint satisfies every constraint, whereas robust-envelope training will keep the measured maximum grid violation at or below zero up to numerical tolerance. As the base learning rate increases, the robust method should show an acceptance transition near the largest rate for which path constraints remain feasible; the estimated threshold should change by less than 20% when increasing $K$ from 8 to 16. The expected trade-off is a lower effective learning rate and fewer transient loss or activation spikes, not necessarily a lower final training loss.