Idea description

Replace an unconstrained linear recurrent update with a two-dimensional oscillator state per hidden feature and use amplitude-dependent damping: negative damping below a target radius and positive damping above it. The cell should preserve phase information over long sequences while preventing hidden-state explosion or collapse.

Formulas

$$\dot{x}_k=\omega_k y_k,\qquad \dot{y}_k=-\omega_k x_k+\mu\left(1-\frac{x_k^2+y_k^2}{R^2}\right)y_k+u_k.$$

$$E_k=\frac{1}{2}(x_k^2+y_k^2),\qquad \dot E_k=\mu\left(1-\frac{x_k^2+y_k^2}{R^2}\right)y_k^2+x_k u_k^{(x)}+y_k u_k^{(y)}.$$

$$\langle\dot E_k\rangle\approx C_k\mu\left(1-\frac{\bar r_k^2}{R^2}\right),\qquad \bar r_k^2=\frac{1}{T}\int_t^{t+T}(x_k^2+y_k^2)\,dt,$$

$$(x_{t+1},y_{t+1})=(x_t,y_t)+h\left(\omega_k y_t,\,-\omega_k x_t+\mu\left(1-\frac{x_t^2+y_t^2}{R^2}\right)y_t\right)+hB_k u_t,$$

Mathematical statement

For oscillator $k$ with state $z_k=(x_k,y_k)\in\mathbb{R}^{2}$ and radius $r_k=\|z_k\|$, use $\dot{x}_k=\omega_k y_k$ and $\dot{y}_k=-\omega_k x_k+\mu(1-r_k^2/R^2)y_k+u_k$, where $\omega_k>0$ is angular frequency, $\mu>0$ is nonlinear gain, $R>0$ is the target amplitude, and $u_k$ is an input drive. Without input, the energy $E_k=(x_k^2+y_k^2)/2$ satisfies $\dot E_k=\mu(1-r_k^2/R^2)y_k^2$. Thus the nonlinear term injects energy when $r_k<R$ and removes energy when $r_k>R$ on average over a cycle. This transfers the paper's invariant amplitude relation $(c_{1m}^{k})^2+(c_{2m}^{k})^2=r^2$ into a recurrent hidden-state constraint.

Implementation notes

1. Integration point: use this transition as the hidden update of an Elman-style RNN or a diagonal state-space model. Represent hidden width $d$ by $d/2$ oscillator pairs $(x_k,y_k)$, with learned input matrix $B$, positive frequencies $\omega_k$, gain $\mu$, and target radius $R$. 2. Pseudocode: compute r2 = x*x + y*y; compute dx = omega*y + Bx(u) and dy = -omega*x + mu*(1-r2/R**2)*y + By(u); update x = x + h*dx and y = y + h*dy. Use RK2 if Euler discretization is unstable. 3. Computed from the paper: the amplitude-dependent gain/loss mechanism and fixed-radius interpretation. Estimated empirically: the cycle-average coefficient $C_k$, the finite-step stability region, and the effect of forcing. Log radius distributions, energy drift, gradient norms, and the recurrent Jacobian spectral radius. 4. First cheap experiment: compare this cell with a vanilla RNN, orthogonal RNN, and GRU on copy-memory, adding, and sequential-MNIST tasks at sequence lengths 100, 500, and 1000. Sweep $h\mu\in[0.01,1]$ and $R\in\{0.5,1,2\}$. The prediction is that the unforced cycle-averaged energy drift changes sign near $\bar r=R$, and hidden radii remain bounded near that value. For the undamped Euler rotation, numerical divergence should begin near $h\omega=2$; the measured boundary should agree within approximately 20%.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a readable Van der Pol oscillator cell with RK4 integration and reproducible numerical checks. The mechanism manifested: energy injection/removal changed sign exactly at r=R, observed mu-scaling matched the predicted slope (0.0768 versus 0.0768), and all tested initial radii converged to R=1.0 with approximately 6e-10 tail variation. However, the delayed-memory readout did not beat the vanilla tanh reservoir (MSE 0.4474 versus 0.3514), so there was no observed accuracy win; the idea's stated forward-Euler rotation boundary near h omega=2 is also incorrect, while the measured RK4 linear boundary was near h omega=2.8.

Agent confidence: 8/10

Baseline: Vanilla tanh reservoir: delayed-memory test MSE 0.351394, correlation 0.102633.
Idea: Van der Pol RK4 cell: delayed-memory test MSE 0.447388, correlation -0.091002; energy-sign means at radii [0.5, 0.8, 1.0, 1.2, 1.5] were [0.09375, 0.1152, approximately 0, -0.3168, -1.40625]; observed and predicted mu slopes were both 0.0768; regulation tail radius was 1.000000001 across initial radii 0.1 to 4.0.

Limitations: Only a small fixed delayed-memory reservoir/readout comparison was tested; no trained Elman/GRU, copy-memory, adding, sequential-MNIST, FLOP-normalized study, learned frequencies, parameter sweeps over R and mu, gradient/Jacobian analysis, or GPU experiment was performed. The Euler stability claim was not confirmed because forward Euler applied to an undamped rotation is expansive for every nonzero step; the reported stability sweep concerns RK4 instead.

How to run: python3 run_experiment.py && /home/maxwelhelp/main/bin/python3 mini_compare.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.18337) Sep 2, 2026

Beats the tuned baseline by 58.4% (p=0.0237, wins 7 / 8 paired seeds; metric: lower is better).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0008
Idea mean: 0.0003
Effect (Δ): -0.0005 (−58.4%; negative = idea better)
Wins: 7 / 8 paired seeds
p-value: 0.0237 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±53.1%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in dynamics track was tested; tabular, vision, and sequence tracks were not applicable to the recurrent stability/control mechanism. The radius signature was measured from one trained model and used a broad boundedness tolerance; it does not establish convergence specifically to R=1. The implementation uses Euler integration rather than RK2/RK4 and keeps frequency, gain, radius, and step size fixed rather than learning or sweeping them.

How to run: python3 bench_vdp.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Worked ✓

Agent confidence: 8/10

The structurally matched actuated-pendulum dynamics benchmark used identical training budgets and a shared learning-rate grid for vanilla RNN and Van der Pol recurrent systems. The idea significantly improved test MSE: mean 0.0003242 versus 0.0007790 for the tuned baseline, with 7/8 paired wins and permutation p=0.02365. The mechanism signature was weak rather than a precise confirmation: the observed trained final radius was 0.2807 versus target R=1.0, although it remained bounded.

Baseline: Vanilla RNN, tuned lr=0.01: test MSE mean 0.0007789819 ± 0.0005449067 across 8 seeds; per-seed [0.0004327, 0.0009103, 0.0018180, 0.0015056, 0.0003418, 0.0005398, 0.0003553, 0.0003285]. Baseline sweep means: lr=0.001 -> 0.0109576, lr=0.003 -> 0.0040046, lr=0.01 -> 0.0011666.
Idea: Van der Pol recurrent cell, best lr=0.01: test MSE mean 0.0003241945 ± 0.0001932752 across 8 seeds; per-seed [0.0002170, 0.0001709, 0.0004196, 0.0006575, 0.0003262, 0.0005628, 0.0001599, 0.0000798]. Paired delta idea-baseline = -0.0004547874, 7/8 wins, permutation p=0.02365.

Benchmark result