△ Mechanism confirmed, baseline not beaten 2026

Frequency-Response Regularization for Neural Dynamics

Implementation & benchmark of arXiv:2609.03929 — Quantifying Power to Voltage and Frequency Dynamics for Oscillation Propagation Assessment

Usefulness7/10

Difficulty6/10

Novelty7/10

Regularization Stability Accuracy Rnn Ssm Optimizer Training dynamics Control theory Dynamical systems Spectral theory Linear algebra Harmonic analysis

Source paper: Quantifying Power to Voltage and Frequency Dynamics for Oscillation Propagation Assessment arXiv:2609.03929 ⓘ · analyzed Sep 4, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Treat the hidden-state update of an RNN, SSM, or neural ODE as a linearized input-output system and penalize its frequency-response peaks. The regularizer targets amplification caused by nonnormal state matrices, which may be large even when all eigenvalues are stable, and therefore controls transient oscillations and long-horizon sensitivity more directly than an eigenvalue-radius penalty.

Formulas

$$\Delta\dot{x}=A\Delta x+B\Delta u,\qquad \Delta y=C\Delta x+D\Delta u$$

$$G(s)=C(sI-A)^{-1}B+D,\qquad S(\omega)=\sigma_{\max}\left(G(j\omega)\right)$$

$$\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{task}}+\lambda_{\mathrm{fr}}\,\tau\log\left[\sum_{\omega\in\Omega}\exp\left(\frac{S(\omega)}{\tau}\right)\right]$$

$$G(z)=C(zI-J)^{-1}B+D,\qquad z=e^{j\theta},\qquad \rho(J)<1$$

Mathematical statement

For a linearization with state perturbation Delta x dot = A Delta x + B Delta u and output perturbation Delta y = C Delta x + D Delta u, the transfer matrix is G(s) = C(sI - A)^(-1)B + D. For a neural dynamical block, A is the Jacobian of the state dynamics with respect to the hidden state, B is the Jacobian with respect to the input, C is the output Jacobian with respect to the hidden state, and D is the direct input-output Jacobian. At frequency omega, define sensitivity S(omega) as the largest singular value of G(j omega). Penalize the maximum or a soft maximum of S over a task-relevant frequency grid. The mechanism is resolvent amplification: even if every eigenvalue of A has negative real part, nonnormal A can produce a large resolvent norm and strong oscillatory transient growth. For a discrete recurrent map, replace s by z on the unit circle and require spectral radius rho(J) less than one for asymptotic stability.

Implementation notes

Integrate this into a small RNN or state-space layer during training. For a continuous neural ODE, choose logarithmically spaced frequencies over the task-relevant band; for an RNN, choose theta values from zero to pi and use z = exp(j theta). On each minibatch, obtain the local Jacobians A or J, B, C, and D using automatic differentiation. For hidden dimensions below about 256, compute exact Jacobians. For larger models, estimate the largest singular value using randomized vectors and three to ten power iterations. For every frequency, solve the linear system (j omega I - A)v = Bq instead of explicitly forming an inverse. Estimate the largest singular value of the resulting transfer action, accumulate the soft-maximum penalty, and backpropagate the task loss plus lambda_fr times this penalty. The paper supplies the state-space transfer construction and frequency-sensitivity interpretation; the neural Jacobians, frequency band, and regularization weight are estimated or selected empirically. Start with a 64-unit tanh RNN on adding and copy tasks, comparing vanilla training, spectral-radius regularization, and frequency-response regularization. Inject sinusoidal hidden-state perturbations after training and measure output amplitude, gradient norms, and long-horizon loss. The predicted signature is that the measured peak frequency agrees with the resolvent peak within one grid interval, increasing lambda_fr reduces the peak monotonically, and long-horizon gradient amplification falls sharply when the peak response is reduced below roughly one to two.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a self-contained discrete-time RNN frequency-response regularizer and reproducible experiment. The toy check clearly reproduced the claimed nonnormal phenomenon: two matrices with identical stable eigenvalues and spectral radius 0.72 had a 17.8x difference in resolvent peak, while the nonnormal matrix showed 10.18x transient gain. In the RNN experiment, frequency regularization reduced the final resolvent peak by 54.4% versus baseline, but slightly worsened task loss by 0.0135, so it controlled sensitivity without demonstrating an accuracy win.

Agent confidence: 8/10

Baseline: final task loss 0.5048; resolvent peak 3.6520; spectral radius 1.7548
Idea: final task loss 0.5182; resolvent peak 1.6644; spectral radius 1.7254; peak reduction 54.4%

Limitations: The training comparison used one fixed seed, a tiny 8-unit tanh RNN, and a short sequence-to-sum task. The regularizer used exact linearization at zero state rather than minibatch local Jacobians, no spectral-radius-regularized control was run, and the trained recurrent matrices remained above unit spectral radius, so this is evidence for peak control rather than a complete stability result.

How to run: python3 frequency_response_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.03929) Sep 4, 2026

No significant difference vs. the tuned baseline (+2.5%, p=0.9833, wins 6 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0014
Idea mean: 0.0014
Effect (Δ): 0 (+2.5%; negative = idea better)
Wins: 6 / 8 paired seeds
p-value: 0.9833 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±73.0%
Mechanism: Confirmed ✓
Practical effect: Inconclusive
Baseline tuning: swept over 6 configs

Limitations:

Only the built-in dynamics track was tested, with 400 training and 200 test examples, 12 epochs, a 12-frequency regularization grid, and 8 paired seeds. The Jacobian signature was evaluated at zero hidden state and zero input rather than across a distribution of rollout states; no continuous-time neural ODE, spectral-radius control baseline, or longer-horizon robustness metric was tested. The regularizer implementation uses exact GRU Jacobians and therefore is costly for larger hidden dimensions.

How to run: python3 bench_frequency_dynamics.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 9/10

Built frequency-response regularization for the built-in controlled-pendulum dynamics track using the shared rnn_small architecture and a custom training loop, with exact local GRU Jacobians and discrete resolvent soft-max penalty. Across 8 paired seeds, the idea had slightly worse MSE (0.0014455 vs 0.0014107; delta +0.0000348), with p=0.98335, so there is no significant task-metric win. The mechanism signature was confirmed: mean trained-model resolvent peak decreased from 1.1065 to 1.0917 (1.34%), but this did not translate into better benchmark accuracy.

Baseline: test MSE mean 0.0014106978, std 0.0003818410; best config lr=0.01, weight_decay=0.0001; 8 per-seed results
Idea: test MSE mean 0.0014454917, std 0.0009746414; best config lr=0.01, lambda_fr=0.0005; 8 per-seed results
Mechanism signature: trained-model local resolvent peak baseline 1.1064540, idea 1.0916701, reduction 1.336%, confirmed=true
Paired delta: idea - baseline = +0.0000347938 MSE
Permutation p: 0.98335

Benchmark result