✗ Failed on benchmark 2026

Hankel Residual Observer

Implementation & benchmark of arXiv:2607.07082 — Model-Free Disturbance Observer with Online Modification: Listening to MFDOOM

Usefulness6/10
Difficulty4/10
Novelty7/10

Source paper: Model-Free Disturbance Observer with Online Modification: Listening to MFDOOM arXiv:2607.07082 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Attach a model-free residual-dynamics observer to a neural multi-step forecaster. Instead of asking the network to relearn persistent periodic or autoregressive disturbances, maintain a Hankel dictionary of recent forecast errors and use ridge reconstruction to predict the next residual sequence online. Add the predicted residual to the network forecast with a confidence-dependent correction gain.

Formulas

$$\mathcal{H}_{L}(x):=\begin{bmatrix}x_{1}&x_{2}&\dots&x_{T-L+1}\\x_{2}&x_{3}&\dots&x_{T-L+2}\\\vdots&\vdots&\ddots&\vdots\\x_{L}&x_{L+1}&\dots&x_{T}\end{bmatrix}$$
$$\begin{bmatrix}D_{p}\\D_{f}\end{bmatrix}\mathbf{g_{d}}=\begin{bmatrix}d_{ini}\\\mathbf{d}\end{bmatrix}$$
$$g_d^*=\arg\min_g\left\|D_pg-d_{ini}\right\|_2^2+\lambda_g\left\|g\right\|_2^2,\qquad \hat{\mathbf d}=D_fg_d^*$$
$$\hat y_{corr}=\hat y+\Gamma\hat d,\qquad \Gamma=\operatorname{diag}(\gamma_1,\ldots,\gamma_{pH})$$

Mathematical statement

For a vector sequence $x_t\in\mathbb{R}^{p}$, the order-$L$ Hankel matrix is $\mathcal{H}_{L}(x)=\begin{bmatrix}x_{1}&x_{2}&\dots&x_{T-L+1}\\x_{2}&x_{3}&\dots&x_{T-L+2}\\\vdots&\vdots&\ddots&\vdots\\x_{L}&x_{L+1}&\dots&x_{T}\end{bmatrix}$. MFDOOM constructs $D_p$ and $D_f$ from prediction-error samples, where $D_p$ contains past windows of length $T_{ini_d}$ and $D_f$ contains their matching future windows of length $H$. Given the latest error history $d_{ini}\in\mathbb{R}^{pT_{ini_d}}$, solve $g_d^*=\arg\min_g\|D_pg-d_{ini}\|_2^2+\lambda_g\|g\|_2^2$ and predict $\hat{\mathbf d}=D_fg_d^*$. The paper corrects predictions using $\mathbf y_{corr}=\mathbf y-\Gamma\mathbf d$, where $\mathbf y$ is the base prediction and $\Gamma$ is a correction-gain matrix. If residuals are stored as actual-minus-predicted, the equivalent implementation is $\hat y_{corr}=\hat y+\Gamma\hat d$. The reconstruction residual $\|D_pg_d^*-d_{ini}\|_2$ supplies a confidence test; disable or attenuate corrections when it exceeds a threshold.

Implementation notes

Integrate the observer after a neural forecaster's multi-step output head and before inference-time prediction is returned or before the forecasting loss is evaluated. Let the network output $\hat y_{t+1:t+H}\in\mathbb{R}^{pH}$. When targets become available, compute the one-step residual $r_t=y_t-\hat y_t$ and append it to a bounded FIFO buffer. Once at least $T_{ini_d}+H$ residual vectors are available, construct dictionary columns from consecutive windows: $D_p[:,j]=[r_j;\ldots;r_{j+T_{ini_d}-1}]$ and $D_f[:,j]=[r_{j+T_{ini_d}};\ldots;r_{j+T_{ini_d}+H-1}]$. Retain the latest $K$ columns and rebuild or rank-update the Gram matrix. At prediction time, form $d_{ini}$ from the latest $T_{ini_d}$ residuals and solve $g=(D_p^TD_p+\lambda_gI)^{-1}D_p^Td_{ini}$ with Cholesky. Compute $\hat d=D_fg$. Define normalized mismatch $q=\|D_pg-d_{ini}\|_2/(\|d_{ini}\|_2+10^{-8})$ and set $\gamma_i=\operatorname{clip}(1-q/\tau,0,1)$, optionally multiplied by a learned horizon decay. Return $\hat y_{corr}=\hat y+\Gamma\hat d$. The Hankel construction and ridge solve come directly from the mathematical method; select $K$, $T_{ini_d}$, $\lambda_g$, $\epsilon$, and $\tau$ on a validation stream. First test a small GRU or TCN on synthetic forecasting with additive sinusoidal disturbances, then Electricity or Traffic. Compare the baseline at equal neural FLOPs and report multi-step RMSE, long-horizon error, observer CPU latency, and memory. The expected signal is lower error on persistent disturbances without degradation on clean data, with corrections automatically suppressed when dictionary mismatch is high.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built a bounded Hankel residual observer with ridge reconstruction and confidence-scaled correction around a small GRU forecaster. Exact period-4 residuals were reconstructed with RMSE 8.6e-13, and disturbed-stream RMSE improved from 0.9907 to 0.8792. Clean-stream RMSE worsened from 0.05314 to 0.05495, so the benefit is promising but not robust across regimes.

Agent confidence
7/10
Baseline
Disturbed RMSE 0.9907; disturbed long-horizon RMSE 1.1564; clean RMSE 0.05314; clean long-horizon RMSE 0.05773.
Idea
Disturbed RMSE 0.8792; disturbed long-horizon RMSE 1.1278; clean RMSE 0.05495; clean long-horizon RMSE 0.06024; p95 observer latency 107.3 ms disturbed and 95.7 ms clean.

Limitations: Only a synthetic AR(1) plus sinusoidal-disturbance stream was tested. No real benchmark, validation-based hyperparameter selection, multi-seed uncertainty, GPU timing, baseline latency, memory, or optimized incremental Gram updates were evaluated.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.07082)

Significantly WORSE than the tuned baseline (+31.8%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark
Sequence modeling
Model
transformer_tiny
Paired seeds
8
Baseline mean
0.5443
Idea mean
0.7173
Effect (Δ)
0.173 (+31.8%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect
±3.8%
Mechanism
Not confirmed ✗
Practical effect
Hurts
Baseline tuning
swept over 3 configs
Limitations:

Only the registered built-in sequence track was tested. No external real-world dataset, dynamics track, observer latency, or memory benchmark was included.

How to run: python3 bench_hankel.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark latest Failed ✗

Agent confidence: 10/10

The Hankel Residual Observer was implemented and evaluated against independently trained transformer_tiny systems on the structurally matched registered sequence track. It significantly worsened test MSE: baseline 0.5443 versus idea 0.7173, paired delta +0.1730 with permutation p=0.0081. The trained-model mechanism signature was not confirmed because residual forecast MSE exceeded the naive residual MSE.

Baseline
Best lr=0.001; mean test MSE 0.5443045375, std 0.0343916344.
Idea
Best lr=0.001; mean test MSE 0.7173206874, std 0.0432494822; paired delta +0.1730161499, p=0.0081, 0/8 wins.

Benchmark result

Significantly WORSE than the tuned baseline (+31.8%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark
Sequence modeling
Model
transformer_tiny
Paired seeds
8
Baseline mean
0.5443
Idea mean
0.7173
Effect (Δ)
0.173 (+31.8%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect
±3.8%
Mechanism
Not confirmed ✗
Practical effect
Hurts
Baseline tuning
swept over 3 configs
Limitations:

Only the registered built-in sequence track was tested. No external real-world dataset, dynamics track, observer latency, or memory benchmark was included.

How to run: python3 bench_hankel.py

Artifacts

Implementation overview ⬇ Download all as ZIP 5 files · code, reports and structured results