✗ Failed on benchmark 2026

FSAL Runge-Kutta Neural Block

Implementation & benchmark of arXiv:2607.06788 — On the order of Runge Kutta methods reusing last stage

Usefulness7/10

Difficulty5/10

Novelty5/10

Architecture Speedup Accuracy Stability Ssm Mlp Transformer Inference speedup Training dynamics Numerical analysis Dynamical systems Approximation theory Linear algebra

Source paper: On the order of Runge Kutta methods reusing last stage arXiv:2607.06788 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a weight-tied residual or neural-ODE stepper with an explicit Runge–Kutta method satisfying the reused-last-stage conditions. The final derivative is evaluated at the exact endpoint and becomes the first derivative of the next step, saving one expensive neural-vector-field call per step while preserving the designed integration order.

Formulas

$$y_{n+1}=y_n+h\sum_{j=1}^{s}b_j f_{n,j},\qquad f_{n,j}=f\left(t_n+c_jh,\ y_n+h\sum_{k<j}a_{jk}f_{n,k}\right).$$

$$c_s=1,\qquad a_{sj}=b_j\quad(j=1,\ldots,s-1),\qquad b_s=0\quad\Longrightarrow\quad f_{n,s}=f(t_{n+1},y_{n+1})=f_{n+1,1}.$$

$$\begin{array}{c|cccc}0&0&0&0&0\\[1pt]1/2&1/2&0&0&0\\[1pt]3/4&0&3/4&0&0\\[1pt]1&2/9&1/3&4/9&0\\\hline&2/9&1/3&4/9&0\end{array}$$

$$\widehat{A}^{(n)}=\frac{1}{n}\begin{pmatrix}A&\\E&A\\eb^T&E&A\\\cdots&\cdots&\ddots&\ddots&\\eb^T&\cdots&eb^T&E&A\end{pmatrix},\qquad \widehat{b}^{(n)}=\frac{1}{n}\begin{pmatrix}b\\b\\\vdots\\b\end{pmatrix}.$$

Mathematical statement

The paper uses the explicit s-stage Runge–Kutta update y_{n+1}=y_n+h\sum_{j=1}^{s}b_j f_{n,j}, where y_n is the state at time t_n, h is the step size, f is the neural vector field, A=(a_{jk}) is strictly lower triangular, b is the output-weight vector, and c is the stage-time vector. The stages are f_{n,j}=f(t_n+c_jh,y_n+h\sum_{k<j}a_{jk}f_{n,k}). A sufficient first-same-as-last, or FSAL, condition is c_s=1, a_{sj}=b_j for j<s, and b_s=0. Under these identities, the last stage is f_{n,s}=f(t_{n+1},y_{n+1}), exactly the first stage needed by the next step, so it can be cached. A concrete third-order four-stage tableau is c=(0,1/2,3/4,1), A_{21}=1/2, A_{32}=3/4, A_{41}=2/9, A_{42}=1/3, A_{43}=4/9, and b=(2/9,1/3,4/9,0)^T. The paper's composed-method matrix formalizes repeated reuse across n steps: \widehat{A}^{(n)}=\frac{1}{n}\begin{pmatrix}A&\\E&A\\eb^T&E&A\\\cdots&\cdots&\ddots&\ddots&\\eb^T&\cdots&eb^T&E&A\end{pmatrix}, with \widehat{b}^{(n)}=\frac{1}{n}(b^T,b^T,\ldots,b^T)^T, where e is the all-ones vector and E=eb^T+M. The transferable property is not the matrix construction itself, but the exact endpoint-stage identity that removes one network evaluation per step.

Implementation notes

Integrate this at the neural-vector-field call site of a weight-tied neural ODE, continuous-depth MLP, SSM, or residual block. Let f_theta(t,y) be one expensive shared network evaluation, y the hidden state, and h the integration step. Use the four-stage tableau in the formula above. On the first step, compute k1=f_theta(t,y), k2=f_theta(t+h/2,y+h*k1/2), k3=f_theta(t+3h/4,y+3h*k2/4), and k4=f_theta(t+h,y+h*(2*k1/9+k2/3+4*k3/9)). Then update y with y <- y+h*(2*k1/9+k2/3+4*k3/9), and cache k4. On each later step, assign k1=cache without calling f_theta; compute k2, k3, and k4 using the same formulas; update y; and replace the cache with k4. Compact pseudocode is: k1 = cache if cache exists else f(t,y); k2=f(t+h/2,y+h*k1/2); k3=f(t+3*h/4,y+3*h*k2/4); k4=f(t+h,y+h*(2*k1/9+k2/3+4*k3/9)); y=y+h*(2*k1/9+k2/3+4*k3/9); cache=k4. The coefficients and endpoint identity come from the paper; wall-clock cost, numerical error, and gradient behavior must be measured empirically. With time conditioning, parameters must be tied across steps and the cached k4 must correspond exactly to the next step's first input (t+h,y). During training, preserve the cache's autograd connection or implement checkpointing that recomputes the producer consistently. First test on a 2D spiral neural ODE and CIFAR-10 with a small weight-tied ResNet, comparing Euler, classical RK4, and FSAL RK3 at equal function-evaluation budgets. Measure state error against a tiny-step RK4 reference, validation accuracy, wall-clock time, peak memory, and gradient norms. Success requires approximately third-order error decay under step halving and about 25% fewer vector-field calls than four-stage RK4 at comparable accuracy.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built and numerically verified a four-stage, third-order FSAL Runge–Kutta neural-ODE-style integrator against classical RK4. The endpoint cache identity held to 1.1e-16, observed convergence slopes were 2.996 for FSAL and 4.011 for RK4, and FSAL used 3n+1 evaluations versus 4n, saving 24.7% at 96 steps. The mechanism manifested clearly, with a 0.792 runtime ratio in the NumPy proxy, though RK4 was substantially more accurate at equal evaluation budgets.

Agent confidence: 9/10

Baseline: Classical RK4: 4n calls, error 1.108e-8 at n=96, convergence slope 4.011, and 0.660 s for 100 n=80 runs.
Equal call example: At approximately 240 calls, FSAL error was 3.353e-6 versus RK4 error 7.277e-8.
Idea: FSAL RK3: 3n+1 calls, error 1.941e-6 at n=96, convergence slope 2.996, endpoint discrepancy <=1.11e-16, 24.7% fewer calls, and 0.523 s for 100 n=80 runs.

Limitations: The experiment used a deterministic NumPy 2D smooth vector field rather than a trained PyTorch neural network, CIFAR-10, GPU execution, autograd, or peak-memory measurement. Runtime is a Python-call proxy and does not establish neural-network wall-clock speedup; equal-call accuracy favored RK4 because it is fourth order.

How to run: python3 fsal_experiment.py > results.json

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.06788) Sep 2, 2026

Significantly WORSE than the tuned baseline (+0.0%, p=0.0466, wins 2 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.003
Idea mean: 0.003
Effect (Δ): 0 (+0.0%; negative = idea better)
Wins: 2 / 8 paired seeds
p-value: 0.0466 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.0%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the registered built-in dynamics track was tested. The finite 8-step rollout used constant control to make exact FSAL reuse valid; time-varying-control cache invalidation, wall-clock speed, peak memory, and a broader equal-FLOP accuracy frontier were not tested.

How to run: python3 bench_fsal.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 9/10

The matched end-to-end dynamics benchmark compared classical RK4 with FSAL RK3 using the same learned neural vector-field architecture, canonical training loop, data, epochs, and learning-rate sweep. FSAL exactly reused the trained model's endpoint derivative and reduced this finite rollout from 32 to 25 calls, but its MSE was marginally worse; the required significant-win condition was not met.

Baseline: RK4 best lr=0.01: mean test MSE 0.003042454074602574 over 8 seeds.
Idea: FSAL RK3 best lr=0.01: mean test MSE 0.0030424724682234228 over 8 seeds; 25 versus 32 vector-field calls per rollout.
Mechanism signature: trained_model_endpoint_max_abs=0.0; observed call saving fraction=0.21875; confirmed=false because finite-horizon savings did not reach the 25% tolerance used.
Paired delta: 1.83936208486557e-08 (positive means FSAL worse); permutation p=0.04655; idea wins 2/8 pairs.

Benchmark result