✗ Failed on benchmark 2026

Cross-prediction determinism gate

Implementation & benchmark of arXiv:2608.07109 — Learning a quantitative criterion for distinguishing chaos from noise

Usefulness7/10

Difficulty4/10

Novelty8/10

Dynamics Stability Generalization Accuracy Rnn Ssm Training dynamics Regularization Dynamical systems Statistics Stochastic processes Control theory

Source paper: Learning a quantitative criterion for distinguishing chaos from noise arXiv:2608.07109 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use a frozen echo-state reservoir and a linear readout to measure whether a time series contains reproducible dynamical structure rather than memorisable temporal correlations. Apply the held-out cross-prediction score as an early-stopping signal, data-quality gate, or regularizer for an RNN or neural state-space forecaster. The mechanism should reduce overfitting to stochastic fluctuations while preserving genuinely predictable chaotic structure.

Formulas

$$\mathbf{r}_t=(1-a)\mathbf{r}_{t-1}+a\tanh(W_r\mathbf{r}_{t-1}+W_{in}u_t+b),\qquad z_t=[\mathbf{r}_t;u_t;1]$$

$$\Delta_\tau u_t=u_{t+\tau}-u_t,\qquad \widehat{\Delta_\tau u}_t=W_o z_t$$

$$W_o=\arg\min_W\sum_{t\in\mathcal{T}_{train}}\|Wz_t-\Delta_\tau u_t\|_2^2+\lambda\|W\|_F^2$$

$$Q_\tau=\rho(\Delta_\tau u,\widehat{\Delta_\tau u})^2,\qquad G_\tau=Q_\tau^{real}-\operatorname{median}_{k=1}^{K}Q_{\tau,k}^{surrogate}$$

Mathematical statement

For observations u_t in R^d, maintain a fixed reservoir state r_t in R^n with leak rate a in (0,1], recurrent matrix W_r, input matrix W_in, bias b, and tanh activation: r_t=(1-a)r_{t-1}+a tanh(W_r r_{t-1}+W_in u_t+b). Define z_t=[r_t;u_t;1] and the future increment Delta_tau u_t=u_{t+tau}-u_t. Fit only a linear readout W_o by ridge regression on training trajectories. Evaluate on an independent trajectory or temporal block using the squared Pearson correlation Q_tau=rho(Delta_tau u, predicted Delta_tau u)^2. Cross-prediction prevents a readout trained on one realization from receiving credit for correlations that do not transfer to another realization. Deterministic or chaotic dynamics should produce a larger Q_tau than matched-correlated-noise and shuffled surrogate data.

Implementation notes

1. Integration point: add this diagnostic beside any sequence forecaster, especially a GRU, RNN, or neural state-space model. Keep the reservoir fixed and do not train it jointly with the main model. Draw W_r and W_in once, rescale W_r to a selected spectral radius, discard an initial washout interval, and fit only the ridge readout on one subset of trajectories. Evaluate on independent trajectories or non-overlapping temporal blocks. Use G_tau as an early-stopping monitor or as a gate that downweights examples whose apparent predictability is not cross-reproducible. 2. Pseudocode: initialize r=0; at each time step update r=(1-a)r+a*tanh(W_r*r+W_in*u+b); store z=[r,u,1] and target u[t+tau]-u[t]; solve W_o=Y X^T (X X^T+lambda I)^(-1) on split A; compute Q_tau on split B; repeat for K phase-randomized, block-shuffled, or matched-autocorrelation surrogate datasets. Reduce the main optimizer learning rate or stop training when forecasting loss improves while G_tau decreases. 3. Taken directly from the paper are the frozen-reservoir architecture, future-increment target, squared Pearson criterion, and cross-prediction protocol. Estimate empirically the reservoir spectral radius, leak rate, ridge coefficient, surrogate null distribution, and threshold for G_tau. 4. First cheap experiment: compare a GRU and neural SSM on Lorenz-63 observations, an AR(1) process with matched autocorrelation, and noisy Lorenz data. Use 5 to 10 reservoir seeds and horizons tau=1,2,4,8,16. The predicted signature is Q_tau(Lorenz)>Q_tau(AR1), while G_tau for shuffled controls is near zero. The real-data Q_tau should decline approximately monotonically with tau, and the real-versus-surrogate gap should remain positive across seeds and exceed the surrogate 95 percent interval.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a frozen leaky-tanh reservoir, ridge readout, squared-Pearson cross-prediction score, AR control, and phase-surrogate evaluation in experiment.py. Three quantitative math predictions were confirmed: reservoir states stayed below 1.0 for spectral radii 0–3, Pearson Q was invariant to prediction scaling with numerical spread below 1e-15, and ridge regularization shrank readout norm from 0.208 to 1.0e-6 while leaving Q unchanged as expected for correlation. The mechanism produced a strong cross-trajectory determinism gap for logistic dynamics versus AR(1) noise at N=3200, but phase-randomized surrogates were not consistently separated at longer horizons, so the result is promising rather than universally validated.

Agent confidence: 8/10

Baseline: Matched AR(1), phi=0.9: Q=[0.039, 0.082, 0.156, 0.278] for tau=[1,2,4,8] at N=3200; phase-surrogate Q=[0.487, 0.488, 0.507, 0.497].
Idea: Independent logistic trajectories: Q=[0.966, 0.738, 0.506, 0.494] for tau=[1,2,4,8] at N=3200; gaps versus AR=[0.927, 0.655, 0.350, 0.216].

Limitations: No GRU or neural state-space early-stopping integration was tested; only scalar logistic dynamics and AR/phase controls were evaluated. The phase-surrogate null was not reliably below the chaotic score at long horizons, and no confidence intervals beyond three reservoir seeds were computed.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.07109) Sep 1, 2026

Significantly WORSE than the tuned baseline (+608.0%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0013
Idea mean: 0.0095
Effect (Δ): 0.0081 (+608.0%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±273.5%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in dynamics track and rnn_small architecture were tested; no GRU/SSM alternatives, longer horizons, noisy dynamics, or regularizer/downweighting variants were evaluated. The mechanism signature used one representative seed rather than confidence intervals across all trained models.

How to run: python3 bench_stage2.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

The cross-prediction gate was implemented as held-out frozen-reservoir-gap early stopping for the built-in dynamics/rnn_small benchmark. Baseline test MSE was 0.0013366373 versus 0.0094634382 for the idea; paired delta was +0.0081268009 with permutation p=0.0081, so the idea was significantly worse. The trained-model signature confirmed the real-versus-shuffled predictability gap, but this did not yield a task-metric win.

Baseline: best lr=0.006; full 8-seed mean test MSE=0.0013366372950258665; std=0.0002929899789605115
Idea: best lr=0.001; full 8-seed mean test MSE=0.009463438211241737; std=0.004041857845276213; paired delta=0.00812680091621587; permutation p=0.0081; verdict=idea worse (significant)

Benchmark result