Idea description

Train a recurrent or neural-ODE state transition with an integral residual instead of matching noisy finite-difference derivatives. Enforce sparse regulator-to-state connectivity with group sparsity, so the model learns a compact dynamical mechanism while avoiding the severe variance amplification caused by estimating derivatives from sampled data.

Formulas

$$\dot{x}(t)=F_{\theta}(x(t),r(t)),\qquad x(t_{j+1})-x(t_j)=\int_{t_j}^{t_{j+1}}F_{\theta}(x(\tau),r(\tau))\,d\tau.$$

$$\mathcal{L}(\theta)=\frac{1}{N}\sum_{j=0}^{N-1}\left\|x_{j+1}-x_j-\Delta t_jF_{\theta}\left(\frac{x_j+x_{j+1}}{2},r_{j+1/2}\right)\right\|_{2}^{2}+\lambda\sum_{g,r}\|W_{gr}\|_{2}.$$

$$\lambda_{\min}\left(\frac{1}{N}\Phi_{S}^{\top}\Phi_{S}\right)>0,\qquad \|\widehat{W}-W^{\star}\|=O\left(\frac{\sigma\sqrt{s\log p/N}}{\lambda_{\min}(\Phi_S^{\top}\Phi_S/N)}\right).$$

Mathematical statement

Let $x(t)\in\mathbb{R}^{d}$ be the hidden state, $F_{\theta}(x(t),r(t))$ the learned vector field, and $0=t_{0}<t_{1}<\cdots<t_{N}$ observation times. The model satisfies $x(t_{j+1})-x(t_j)=\int_{t_j}^{t_{j+1}}F_{\theta}(x(\tau),r(\tau))\,d\tau$. With piecewise-constant or interpolated observations, fit $\theta$ by minimizing the integral residual plus a group-lasso penalty on regulator-to-state blocks $W_{gr}$. If $\Phi$ is the design matrix formed from integrated candidate features and $S$ is the true active support, sparse recovery is controlled by excitation of active directions, for example $\lambda_{\min}(\Phi_S^{\top}\Phi_S/N)>0$, with estimation error decreasing approximately as $O(\sigma\sqrt{s\log p/N}/\lambda_{\min})$, where $\sigma$ is observation noise, $s=|S|$, and $p$ is the number of candidate couplings.

Implementation notes

Integrate this at the transition-learning component of a recurrent state-space model or neural ODE. Let the hidden state be $x_j$, let $r_j$ contain selected input or regulator channels, and parameterize the vector field as a small MLP with explicit linear regulator-to-state blocks $W_{gr}$. Apply group lasso to each block so an entire edge is removed rather than only individual scalar weights. For each minibatch trajectory, compute the midpoint vector field $\hat f_j=F_\theta((x_j+x_{j+1})/2,(r_j+r_{j+1})/2)$, form $L=\sum_j\|x_{j+1}-x_j-\Delta t_j\hat f_j\|^2/N+\lambda\sum_{g,r}\|W_{gr}\|_2$, backpropagate, and use Adam followed by proximal group soft-thresholding. Compute midpoint integrals cheaply; for higher accuracy replace them with two- or four-point quadrature. Estimate the active Gram eigenvalue from the minibatch feature matrix and estimate noise from held-out residuals. First test on 1,000 trajectories of a 20-dimensional sparse tanh RNN with 10% active edges, Gaussian observation noise, and irregular sampling. Compare finite-difference regression, ordinary one-step prediction, and integral training. The prediction is that finite-difference error scales approximately as $\sigma^2/\Delta t^2$, while integral training is substantially less sensitive; support recovery should fail sharply as the empirical active Gram eigenvalue approaches zero.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

The MVP confirmed the predicted derivative-noise amplification: integral increment variance stayed near 0.0098 while finite-difference variance followed 0.0098 divided by delta-t squared. Support recovery degraded when the active Gram minimum eigenvalue approached zero. The integral method achieved a small clean-increment MSE improvement, but not a decisive win.

Agent confidence: 7/10

Baseline: clean increment MSE 2.6686e-5; support F1 1.00
Idea: clean increment MSE 2.5573e-5; support F1 1.00; at rho=0.99, active Gram minimum eigenvalue 0.00664 and support F1 0.50

Limitations: The test used a linear toy sparse-regression system rather than a tanh RNN or neural ODE. It did not test Adam, minibatching, higher-order quadrature, repeated seeds, confidence intervals, or runtime. The support-recovery trend was qualitative and non-monotonic at intermediate correlations.

How to run: python3 integral_sparse_dynamics.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.09722) Sep 2, 2026

No significant difference vs. the tuned baseline (+30.0%, p=0.3185, wins 3 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0103
Idea mean: 0
Effect (Δ): 0.0031 (+30.0%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.3185 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±64.0%
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 6 configs

Limitations:

The completed run used 400 training examples, 150 test examples, and 6 epochs because the initial larger run exceeded the 40-minute limit. The implementation used observed prefix-theta differences as an integral-residual surrogate rather than a separately parameterized vector field with explicit midpoint quadrature; higher-order quadrature and larger-scale training were not tested.

How to run: python3 bench_stage2.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Built and ran Stage-2 on the structurally matched registered dynamics track with eight paired seeds, a tuned baseline sweep, equal learning-rate search space, and a trained-model mechanism signature. The integral-residual system was worse than the tuned endpoint-MSE baseline (delta +0.00310189, p=0.31845), and the quantitative signature prediction was not confirmed. Therefore there is no observed benchmark win.

Baseline: mean test MSE 0.01033135, std 0.00646953; best lr=0.01, weight_decay=0.0001
Idea: mean test MSE 0.01343324, std 0.00623061; best lr=0.01, lam=0.0001; paired delta +0.00310189; 3/8 wins; permutation p=0.31845

Benchmark result