RC-Mode Response Regularizer

Attach the regularizer to a network during training, evaluating it once every M steps to control overhead. Inject small sinusoidal perturbations into the learning rate or update magnitude using \(\theta_{t+1}=\theta_t-\eta_t[1+\epsilon\sin(\omega t)]g_t\). Use a normalized Gaussian direction if a parameter projection is needed, and measure the scalar response as training loss, validation loss, gradient norm, or projected displacement. Estimate \(\widehat H(i\omega_j)\) with lock-in Fourier correlations over at least four probe periods. For the first MVP, do not differentiate through the estimator: compute the response penalty after each probe window and use it to rescale subsequent updates. Fit positive \(\tau_k\) and residues when mode interpretation is useful; otherwise use measured squared amplitudes directly. The paper supplies the parallel relaxation-mode decomposition, while response amplitudes, probe frequencies, mode count, and target \(\tau_0\) are empirical. First experiment: train a small CNN on CIFAR-10, comparing cross-entropy alone with the response penalty at matched compute and several \(\beta\) values. The prediction is that high-frequency gain decreases monotonically with \(\beta\), oscillatory loss disappears near the fixed-learning-rate boundary, and robustness to a twofold learning-rate perturbation improves. Also test whether the dominant fitted pole remains stable and whether loss autocorrelation decays faster than in the unregularized baseline.