✗ Failed on benchmark 2026

Knieper Rollout Stability Metric

Implementation & benchmark of arXiv:2608.13534 — Surfaces with nonpositive magnetic curvature

Usefulness7/10

Difficulty3/10

Novelty6/10

Dynamics Stability Accuracy Rnn Ssm World model Regularization Dynamical systems Geometry Control theory Optimization

Source paper: Surfaces with nonpositive magnetic curvature arXiv:2608.13534 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace pointwise hidden-state distance penalties with a trajectory metric that measures the largest discrepancy over a short rollout. This directly controls transient amplification: two nearly identical states are considered unstable if their predicted trajectories separate at any intermediate time, even when they happen to reconverge at the final step.

Formulas

$$d_K(v,w)=\max_{t\in[0,1]}d\bigl(\gamma_v(t),\gamma_w(t)\bigr).$$

$$D_H(x,y)=\max_{0\le k\le H}\left\|x_k-y_k\right\|_2,\qquad x_{k+1}=F_\theta(x_k,u_k),\quad y_{k+1}=F_\theta(y_k,u_k).$$

$$\mathcal L_{\mathrm{roll}}=\frac{1}{B}\sum_{b=1}^{B}\frac{D_H(x^{(b)},x^{(b)}+\epsilon^{(b)})}{\|\epsilon^{(b)}\|_2+\varepsilon},\qquad \mathcal L=\mathcal L_{\mathrm{task}}+\lambda\mathcal L_{\mathrm{roll}}.$$

Mathematical statement

For a magnetic-flow state v, let $\gamma_v(t)$ denote its trajectory and define the Knieper metric $d_K(v,w)=\max_{t\in[0,1]}d(\gamma_v(t),\gamma_w(t))$. The paper proves that this is a metric and is uniformly equivalent to the Sasaki state-space metric. In a neural dynamical system $x_{k+1}=F_\theta(x_k,u_k)$, use the adapted finite-horizon metric $D_H(x,y)=\max_{0\le k\le H}\|F_\theta^{(k)}(x)-F_\theta^{(k)}(y)\|_2$, where $F_\theta^{(k)}$ is the k-step rollout. Penalizing $D_H$ for nearby perturbed states suppresses transient growth and approximates a finite-time Lipschitz bound.

Implementation notes

Integrate this into the training loss of a GRU, diagonal SSM, or neural ODE used for next-step prediction. For each minibatch sequence, save the hidden or latent state at time k, create a perturbed copy with $y_0=x_0+\epsilon$, and run both copies for H shared-input steps. Pseudocode: x=x0; y=x0+eps; maxsep=0; for k in range(H+1): maxsep=max(maxsep,norm(x-y)); x=F_theta(x,u[k]); y=F_theta(y,u[k]); loss=task_loss+lambda*maxsep/(norm(eps)+eps0); loss.backward(). Use the exact maximum for small H; for long H use a detached running maximum or a soft maximum $\tau\log\sum_k\exp(\|x_k-y_k\|/\tau)$. The paper supplies the max-over-time construction and its metric interpretation; the perturbation radius, horizon H, and coefficient lambda are empirical. First run a copy task and sequential-MNIST experiment with a vanilla GRU, spectral-normalized GRU, and this regularizer. Sweep learning rate and lambda, and measure the finite-time gain $G_H=\max_{k\le H}\|\delta x_k\|/\|\delta x_0\|$. The prediction is that the regularizer lowers the upper tail of $G_H$, especially at intermediate times, and that long-horizon loss becomes sharply worse when the unregularized model crosses $G_H\approx 1$ persistently. A useful target is at least a 30% reduction in median $G_H$ at matched one-step accuracy, with no more than 10% task-loss degradation.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built the exact max-over-time rollout regularizer for a small GRU and a deterministic toy verification suite. The scalar stability boundary occurred exactly at |a|=1, the nilpotent example showed D_H detecting intermediate amplification that final-step distance misses, and the nonnormal-map gain matched the predicted K-dependent scaling to numerical precision. In the GRU mini-experiment, rollout regularization reduced median G8 from 0.5372 to 0.0972 (81.9% reduction) while loss changed from 0.94443 to 0.94498 (+0.058%), so the proposed effect was observed.

Agent confidence: 8/10

Baseline: Toy: scalar gains [1.0, 1.0, 1.0, 1.0, 1.08286, 1.47746, 4.29982]; GRU task loss 0.9444323, median G8 0.5372056.
Idea: Toy: all predicted gains matched observed values to floating-point precision; transient final-step gain stayed 0 while D_H reached 4.0 at K=4; GRU lambda=0.03 task loss 0.9449787, median G8 0.0971686, an 81.9% gain reduction.

Limitations: Only one small synthetic next-step prediction task, one random seed, one lambda, one horizon, and 100 optimization steps were tested; no sequential-MNIST, spectral-normalized GRU, long-horizon generalization, FLOP-normalized comparison, or multi-seed statistics were run. CUDA was attempted but failed with a shared-environment cuDNN host-allocation error, so the GRU comparison used CPU fallback.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.13534) Sep 2, 2026

No significant difference vs. the tuned baseline (+5.3%, p=0.3862, wins 4 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0486
Idea mean: 0.0512
Effect (Δ): 0.0026 (+5.3%; negative = idea better)
Wins: 4 / 8 paired seeds
p-value: 0.3862 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in dynamics track was tested, with 5 epochs, 400 training samples, 200 test samples, hidden size 16, and three lambda values.

How to run: python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

On the registered dynamics track with matched rnn_small GRUs, baseline test MSE was 0.0486428 and the best rollout-regularized model was 0.0512108. The paired delta was +0.0025680 with permutation p=0.3862, so there was no significant win; observed mean G_H reduction was only 1.61%.

Baseline: mean test MSE 0.0486427885, best lr 0.006
Idea: mean test MSE 0.0512108102, lr 0.006 lambda 0.01, mean G_H reduction 1.61%

Benchmark result