✗ Failed on benchmark 2026

Residual-Gated Streaming Adaptation

Implementation & benchmark of arXiv:2608.21302 — Fault Diagnosis of Dynamic Systems Under Unknown Operating Conditions: A Condition-Guided Selective Adaptation Approach

Usefulness8/10
Difficulty5/10
Novelty6/10

Source paper: Fault Diagnosis of Dynamic Systems Under Unknown Operating Conditions: A Condition-Guided Selective Adaptation Approach arXiv:2608.21302 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use the condition discriminator's residual and predictive variance to decide which unlabeled streaming samples may update a model at deployment. Only samples whose condition prediction is both calibrated and close to the currently expected condition are admitted, preventing unreliable operating regimes from causing catastrophic test-time drift.

Formulas

$$r_t=\frac{c_t-\hat c_t}{\sqrt{v_t+\epsilon}},\qquad \hat c_t=\mu_\psi(F_\theta(x_t)),\qquad v_t=\sigma_\psi^2(F_\theta(x_t)).$$
$$g_t=\mathbf{1}\{|r_t|\leq\tau_r\}\mathbf{1}\{v_t\leq\tau_v\},\qquad q_t=\exp\left(-\frac{r_t^2}{2}\right)(v_t+\epsilon)^{-1/2}.$$
$$\alpha_{t+1}=\alpha_t-\eta_a g_tq_t\nabla_\alpha L_u(x_t),\qquad \bar c_{t+1}=\rho\bar c_t+(1-\rho)\hat c_t.$$
$$\|\alpha_T-\alpha_0\|\leq\eta_a\sum_{t=1}^{T}g_tq_t\|\nabla_\alpha L_u(x_t)\|.$$

Mathematical statement

Let $x_t$ be a streaming sample, $\hat c_t=\mu_\psi(F_\theta(x_t))$, $v_t=\sigma_\psi^2(F_\theta(x_t))$, and $r_t=(c_t-\hat c_t)/\sqrt{v_t+\epsilon}$ when a condition measurement $c_t$ is available. If $c_t$ is unavailable, use the residual between the current condition estimate and an exponential moving prediction $\bar c_t$: $r_t=(\hat c_t-\bar c_t)/\sqrt{v_t+\epsilon}$. Define an admission score $q_t=\exp(-r_t^2/2)/\sqrt{v_t+\epsilon}$ or the hard gate $g_t=\mathbf{1}(|r_t|\leq\tau_r, v_t\leq\tau_v)$. For admitted samples, update only adaptation parameters $\alpha$ such as normalization statistics, a low-rank adapter, or the final classifier layer using an unsupervised consistency or entropy loss $L_u$. The update is $\alpha_{t+1}=\alpha_t-\eta_a g_tq_t\nabla_\alpha L_u(x_t)$. The cumulative update is bounded by the sum of admitted weighted gradients, providing a directly measurable drift-control quantity.

Implementation notes

1. Exact integration point: freeze the backbone and condition discriminator after offline training; expose only LayerNorm or BatchNorm affine parameters, a small LoRA adapter, or the final linear fault head to online updates. For each incoming sample, compute the fault posterior, condition mean, condition variance, and a residual against either a measured operating condition or an EMA forecast. 2. Pseudocode: initialize $\bar c$ from the offline condition distribution; for each $x_t$, compute $(p_t,\hat c_t,v_t)$, set $r_t=(\hat c_t-\bar c)/\sqrt{v_t+\epsilon}$ if no sensor condition exists, set $g_t=1[|r_t|<\tau_r\land v_t<\tau_v]$, compute entropy or temporal-consistency loss $L_u=H(p_t)$ or $\|p_t-p_{t-1}\|^2$, and update $\alpha\leftarrow\alpha-\eta_a g_tq_t\nabla_\alpha L_u$. Clip the update norm and update $\bar c$. 3. Taken from the paper: continuous condition residuals as reliability signals and selective online adaptation. Estimated empirically: $\tau_r$, $\tau_v$, EMA coefficient $\rho$, and whether entropy minimization or consistency is safer. 4. First cheap experiment: use a frozen MLP or 1-D CNN on a rotating-machinery benchmark or synthetic signals with slowly drifting speed. Train offline on one speed range and stream through a held-out range with labels hidden during adaptation. Compare no adaptation, adapt-every-sample, confidence-gated adaptation, and residual-gated adaptation. Sweep $\tau_r\in\{1,1.5,2,2.5,3\}$. The predicted signature is lower parameter drift and lower post-adaptation loss for accepted samples; samples with $|r|>3$ should produce disproportionately large error and drift. Measure gate acceptance, cumulative admitted weight, parameter drift, accuracy, and catastrophic-drift frequency. The method is supported if drift scales approximately linearly with $\sum_tg_tq_t$ and accepted samples have lower subsequent loss than rejected samples.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a reproducible residual/variance-gated streaming adaptation MVP with analytical and synthetic checks. The mechanism manifested: Gaussian gate acceptance matched the predicted 2Phi(tau)-1 law, the cumulative update bound held, and drift scaled linearly with admitted weighted update mass (R²=1.0). In the toy stream, gating reduced bias drift versus adapting every sample (0.0690 vs 0.0851) while preserving baseline accuracy, but it did not produce an accuracy improvement.

Agent confidence
8/10
Baseline
No adaptation: accuracy 0.728, bias drift 0.0000
Idea
Residual-gated adaptation: accuracy 0.728, acceptance 0.543, weighted mass 745.41, bias drift 0.0690; all-sample adaptation had accuracy 0.727 and drift 0.0851. Math checks: bound 0.6766 <= 12.9421, Gaussian acceptance error <= 4.6e-4, drift scaling R²=1.0.

Limitations: Only a synthetic one-dimensional logistic stream was tested, with entropy minimization updating a single classifier bias. No real rotating-machinery dataset, calibrated learned variance model, measured-condition branch, temporal-consistency loss, multiple random seeds, catastrophic-drift frequency analysis, or compute/FLOP comparison was evaluated.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.21302)

Ahead of the baseline (−0.0%) but not statistically significant (p=0.339, wins 5 / 8 paired seeds).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
1.532
Idea mean
1.5318
Effect (Δ)
-0.0003 (−0.0%; negative = idea better)
Wins
5 / 8 paired seeds
p-value
0.339 (permutation test, 20 000 shuffles)
Smallest detectable effect
n/a
Mechanism
Not confirmed ✗
Practical effect
No effect
Baseline tuning
swept over 0 configs
Limitations:

Only the built-in dynamics pendulum track was tested. The condition proxy was derived from operating-state features, predictive variance was a global offline residual variance rather than a separately calibrated discriminator, and no measured-condition branch, real rotating-machinery dataset, multi-step online deployment horizon, or compute comparison was evaluated.

How to run: python3 bench_residual_gate.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark latest Failed ✗

Agent confidence: 9/10

Implemented residual/variance-gated streaming adaptation on the structurally matched dynamics track using the shared rnn_small architecture and paired 8-seed protocol. The best gated configuration achieved mean test MSE 1.531768 versus the tuned all-sample baseline 1.532036, but the paired permutation test was not significant (delta_mean=-0.000268, p=0.339). All samples were admitted and drift did not correlate with weighted mass (R2=0.0042), so the proposed effect was not demonstrated.

Baseline
Tuned all-sample consistency adaptation; mean MSE 1.5320357531, std 0.0889650146, 8 seeds.
Idea
Residual-gated adaptation; mean MSE 1.5317680091, std approximately 0.089057, paired delta -0.0002677441, 5/8 wins, permutation p=0.339, acceptance 1.0.
Mechanism
Drift-bound held on every seed, but mass-drift R2=0.0042226 and confirmed=false.

Benchmark result

Ahead of the baseline (−0.0%) but not statistically significant (p=0.339, wins 5 / 8 paired seeds).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
1.532
Idea mean
1.5318
Effect (Δ)
-0.0003 (−0.0%; negative = idea better)
Wins
5 / 8 paired seeds
p-value
0.339 (permutation test, 20 000 shuffles)
Smallest detectable effect
n/a
Mechanism
Not confirmed ✗
Practical effect
No effect
Baseline tuning
swept over 0 configs
Limitations:

Only the built-in dynamics pendulum track was tested. The condition proxy was derived from operating-state features, predictive variance was a global offline residual variance rather than a separately calibrated discriminator, and no measured-condition branch, real rotating-machinery dataset, multi-step online deployment horizon, or compute comparison was evaluated.

How to run: python3 bench_residual_gate.py

Artifacts

Implementation overview ⬇ Download all as ZIP 7 files · code, reports and structured results