✗ Failed on benchmark 2026

Reachability-Certified STL Neural ODE Training

Implementation & benchmark of arXiv:2607.08899 — Learning-enabled Parameter Synthesis for Nonlinear Systems from Signal Temporal Logic

Usefulness8/10

Difficulty6/10

Novelty8/10

Dynamics Stability Generalization Accuracy Training dynamics Loss Rnn Ssm Control theory Dynamical systems Optimization Probability

Source paper: Learning-enabled Parameter Synthesis for Nonlinear Systems from Signal Temporal Logic arXiv:2607.08899 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Train a neural ODE or continuous-time recurrent model directly against STL robustness, while requiring the resulting trajectory tube to satisfy the specification for every initial hidden state in a bounded set. Differentiable robustness provides an optimization objective, and interval, zonotope, or other set-based reachability provides a post-update certificate that prevents success caused by a narrow nominal trajectory.

Formulas

$$\dot{x}=f_{\theta}(x,u,t),\qquad x(0)\in\mathcal{X}_0,\qquad \mathcal{R}_{[0,T]}(\theta)=\{(x(t;x_0,\theta),t):t\in[0,T],\ x_0\in\mathcal{X}_0\}.$$

$$\rho(\mathbf{F}_{[a,b]}\varphi,\xi,t)=\max_{s\in[t+a,t+b]}\rho(\varphi,\xi,s),\qquad \rho(\varphi\wedge\psi,\xi,t)=\min\{\rho(\varphi,\xi,t),\rho(\psi,\xi,t)\}.$$

$$\widehat{\mathcal{R}}_t(\theta)\supseteq\mathcal{R}_t(\theta),\qquad \underline{\rho}_{\mathrm{reach}}(\theta)=\inf_{x_0\in\mathcal{X}_0}\rho(\varphi,\xi(\cdot;x_0,\theta),0),$$

$$\theta_{k+1}=\theta_k-\eta\nabla_{\theta}\left[-\rho_{\mathrm{soft}}(\varphi,\xi_{\theta},0)+\lambda\|\theta-\theta_0\|_2^2\right].$$

Mathematical statement

The learned dynamics are $\dot{x}=f_{\theta}(x,u,t)$, where $x$ is the hidden or physical state, $u$ is an input, $t$ is time, and $\theta$ are network parameters. The initial state belongs to $\mathcal{X}_0$. For predicate robustness $\rho(\mu,\xi,t)=\mu(\xi(t),t)$, STL robustness recursively uses negation, conjunction, and temporal extrema: $\rho(\neg\varphi)=-\rho(\varphi)$, $\rho(\varphi\wedge\psi)=\min(\rho(\varphi),\rho(\psi))$, and $\rho(\mathbf{F}_{[a,b]}\varphi,t)=\max_{s\in[t+a,t+b]}\rho(\varphi,s)$. A sound reachable-set overapproximation $\widehat{\mathcal{R}}_t(\theta)$ contains all states generated by $x_0\in\mathcal{X}_0$. If $\underline{\rho}_{\mathrm{reach}}(\theta)$ is the robustness lower bound computed on these sets, then $\underline{\rho}_{\mathrm{reach}}\geq0$ certifies satisfaction for every allowed initial state.

Implementation notes

1. Integration point: use a neural ODE, continuous-time RNN, or latent state-space model whose hidden state follows $\dot{x}=f_\theta(x,u,t)$. Specify bounded initial hidden states $\mathcal{X}_0$, bounded input disturbances, and STL predicates on outputs or states, such as safety bounds, eventual target reaching, or persistence requirements. 2. Pseudocode: initialize theta; sample several initial states and disturbances; integrate trajectories; compute differentiable STL robustness using log-sum-exp soft-min and soft-max; backpropagate negative robustness plus ordinary task loss; update theta. Every K updates, propagate the whole initial set with interval arithmetic, zonotopes, or a first-order affine tube. Evaluate each predicate on every time slice and recursively compute the lower robustness bound. Accept a checkpoint only when the bound is nonnegative; otherwise continue optimization or add the violating predicate and time interval to the loss. 3. Reachability and the lower-bound acceptance implication come from the paper's mechanism. Gradients, smooth robustness, and tube tightness are estimated numerically. 4. First experiment: train a two-dimensional neural ODE initialized over $[-0.2,0.2]^2$, using a damped oscillator dataset and compare MSE-only, sampled-STL, and certified-STL training. Test $\mathbf{F}_{[1,3]}(x_1\geq1)\wedge\mathbf{G}_{[0,4]}(x_2\leq1.2)$. The predicted signature is $\underline{\rho}_{\mathrm{reach}}\geq0$ for accepted models, with every sampled robustness above the certified bound up to numerical tolerance. Increasing the initial-set width should make the certified margin cross zero before nominal performance fails.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a differentiable continuous-time ODE STL-training MVP with soft robustness, an exact sound interval tube, and certified training. The quantitative mechanism checks matched predictions: tube containment held for all tested widths, certified robustness scaled with fitted slope -1.000 versus predicted -1, and the zero-margin width was 1.1554 versus predicted 1.1554; brute-force bound error was 0. Certified training increased the learned rate and retained positive margins, although this simplified system did not produce nominal-training failure within the tested width range.

Agent confidence: 8/10

Baseline: Widths 0.05, 0.30, 0.50, 1.00, 1.50: certified margins 1.100, 0.977, 0.863, 0.598, 0.343; actual dense-sample minima 1.025, 0.990, 0.882, 0.616, 0.359.
Idea: Widths 0.05, 0.30, 0.50, 1.00, 1.50: learned rates 0.846, 0.973, 1.048, 1.230, 1.428 and certified margins 1.100 for every width; dense-sample minima were 1.025 throughout. Verification containment was 5/5, fitted uncertainty-margin slope was -1.000, threshold prediction error was 0, and maximum brute-force bound error was 0.

Limitations: The experiment uses a deliberately simple analytically solvable 2D ODE rather than a general neural ODE, and its interval reachability is exact rather than a challenging interval/zonotope implementation. It does not test noisy inputs, nonlinear learned vector fields, larger STL formulas, computational overhead, CUDA, or a dataset/task MSE tradeoff; the baseline also remained certified-positive over the tested widths.

How to run: python3 reachability_stl_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.08899) Sep 2, 2026

Significantly WORSE than the tuned baseline (+248.8%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0004
Idea mean: 0.0014
Effect (Δ): 0.001 (+248.8%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±103.7%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

The intervention uses endpoint perturbation surrogates rather than a full nonlinear interval/zonotope ODE reachability engine, and the built-in dynamics benchmark predicts a scalar future angle rather than directly evaluating an STL formula. Only the supplied 400-training/200-test dynamics subset, 20 epochs, one perturbation radius, and three learning rates were tested; no larger neural ODE or formal post-update certificate was evaluated.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 10/10

Implemented reachability-inspired certified training on the structurally matched dynamics track using the shared rnn_small architecture and a bounded theta/omega perturbation loss. The interval sanity check passed with containment true for all widths and fitted margin slope -1.0, but the trained-system signature prediction was not confirmed: the idea's mean tube degradation was 0.00930 versus 0.00874 for baseline. On independent test MSE, the idea was significantly worse (0.0014166 versus 0.0004061; paired delta +0.0010105, permutation p=0.0081), so there is no demonstrated transfer win.

Baseline: Best Adam lr=0.01 from sweep {0.001, 0.003, 0.01}; 8-seed test MSE mean 0.0004061429, std 0.0001787820.
Idea: Best certified-loss lr=0.003 from the same 3-setting union; 8-seed test MSE mean 0.0014166450, std 0.0004698827; paired delta +0.0010105021, p=0.0081, 0/8 wins.
Math check: map[containment:[true true true true] fitted_width_slope:-1.0000000000000002 max_endpoint_error:0 predicted_slope:-1]
Mechanism signature: map[baseline_tube_gap:0.0087365526 confirmed:false idea_tube_gap:0.0092958417]

Benchmark result