# Frequency-Response Regularization for Neural Dynamics

- ID: 3155
- Canonical URL: https://synthcore.org/idea/3155/frequency-response-regularization-for-neural-dynamics
- API JSON: https://synthcore.org/api/idea/3155.json
- API Markdown: https://synthcore.org/api/idea/3155.md
- Verification status: unverified
- Source: [arXiv:2609.03929](https://arxiv.org/abs/2609.03929)
- Category: regularization
- Solves: stability, accuracy
- ML areas: rnn, ssm, optimizer, training-dynamics
- Math tags: control-theory, dynamical-systems, spectral-theory, linear-algebra, harmonic-analysis
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Treat the hidden-state update of an RNN, SSM, or neural ODE as a linearized input-output system and penalize its frequency-response peaks. The regularizer targets amplification caused by nonnormal state matrices, which may be large even when all eigenvalues are stable, and therefore controls transient oscillations and long-horizon sensitivity more directly than an eigenvalue-radius penalty.

## Mathematical statement

For a linearization with state perturbation Delta x dot = A Delta x + B Delta u and output perturbation Delta y = C Delta x + D Delta u, the transfer matrix is G(s) = C(sI - A)^(-1)B + D. For a neural dynamical block, A is the Jacobian of the state dynamics with respect to the hidden state, B is the Jacobian with respect to the input, C is the output Jacobian with respect to the hidden state, and D is the direct input-output Jacobian. At frequency omega, define sensitivity S(omega) as the largest singular value of G(j omega). Penalize the maximum or a soft maximum of S over a task-relevant frequency grid. The mechanism is resolvent amplification: even if every eigenvalue of A has negative real part, nonnormal A can produce a large resolvent norm and strong oscillatory transient growth. For a discrete recurrent map, replace s by z on the unit circle and require spectral radius rho(J) less than one for asymptotic stability.

## Key formulas

- $$\Delta\dot{x}=A\Delta x+B\Delta u,\qquad \Delta y=C\Delta x+D\Delta u$$
- $$G(s)=C(sI-A)^{-1}B+D,\qquad S(\omega)=\sigma_{\max}\left(G(j\omega)\right)$$
- $$\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{task}}+\lambda_{\mathrm{fr}}\,\tau\log\left[\sum_{\omega\in\Omega}\exp\left(\frac{S(\omega)}{\tau}\right)\right]$$
- $$G(z)=C(zI-J)^{-1}B+D,\qquad z=e^{j\theta},\qquad \rho(J)<1$$

## Implementation notes

Integrate this into a small RNN or state-space layer during training. For a continuous neural ODE, choose logarithmically spaced frequencies over the task-relevant band; for an RNN, choose theta values from zero to pi and use z = exp(j theta). On each minibatch, obtain the local Jacobians A or J, B, C, and D using automatic differentiation. For hidden dimensions below about 256, compute exact Jacobians. For larger models, estimate the largest singular value using randomized vectors and three to ten power iterations. For every frequency, solve the linear system (j omega I - A)v = Bq instead of explicitly forming an inverse. Estimate the largest singular value of the resulting transfer action, accumulate the soft-maximum penalty, and backpropagate the task loss plus lambda_fr times this penalty. The paper supplies the state-space transfer construction and frequency-sensitivity interpretation; the neural Jacobians, frequency band, and regularization weight are estimated or selected empirically. Start with a 64-unit tanh RNN on adding and copy tasks, comparing vanilla training, spectral-radius regularization, and frequency-response regularization. Inject sinusoidal hidden-state perturbations after training and measure output amplitude, gradient norms, and long-horizon loss. The predicted signature is that the measured peak frequency agrees with the resolvent peak within one grid interval, increasing lambda_fr reduces the peak monotonically, and long-horizon gradient amplification falls sharply when the peak response is reduced below roughly one to two.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.