Moment-Sharp Spectral-Norm Control

(1) Integration point: apply this to each linear or convolutional layer during training, using the squared singular-value spectrum of W. Use K=2 or K=3 initially. The layer can either be rescaled by the computed bound or receive a soft penalty on the excess bound. For a target squared spectral radius q, use W\leftarrow W\sqrt{q/(U_K+\delta)} when U_K>q, or add L_spec=\alpha\max(0,U_K-q)^2 to the loss.

(2) Pseudocode:

for each monitored layer W:
    A(v) = W.T @ (W @ v)                 # convolution: implicit W and W.T
    m1 = HutchinsonTrace(A, R)            # average v_r.T A(v_r)
    m2 = HutchinsonTrace(A @ A, R)
    optionally m3 = HutchinsonTrace(A @ A @ A, R)
    best = +infinity
    for r in 1..K:
        for integer multiplicities n_1+...+n_r=d:
            solve z_j >= eps from sum_j n_j*z_j^k = m_k, k=1..K
            if residual <= tolerance:
                best = min(best, max_j z_j)
    U = best
    W = W * sqrt(q/(U+delta)) if U > q

For K=2, solve the two moment equations for each multiplicity pair directly or with a two-variable Newton solve. For K=3, enumerate multiplicity triples and use multistart Newton; retain only positive roots satisfying the moment residual tolerance. Sort z_j and take the largest feasible value. The paper's mathematics supplies the moment-constrained extremization and the at-most-K-distinct-values reduction; Hutchinson trace estimates, Newton initialization, and the rescaling rule are engineering adaptations. Estimate d from the number of input features and use R=4--16 Rademacher probes. Maintain exponential moving averages of m_k to reduce stochastic root failures.

(3) Cheap first experiment: train a 6-layer MLP and a small ResNet on CIFAR-10 with identical AdamW settings. Compare ordinary weight decay, one-step power-iteration spectral normalization, and K=2/K=3 moment-sharp control at matched wall-clock and FLOPs. Measure training-loss descent, validation accuracy, gradient-norm spikes, estimated true spectral norm from 20-step power iteration, and overhead. Success means the bound tracks the true norm with substantially less bias than trace-only or AM–GM control, reduces instability at the same target q, or reaches the same validation accuracy with fewer power iterations. Also test inference-preserving use: freeze the trained model and compare robustness to input perturbations and activation overflow.