△ Mechanism confirmed, baseline not beaten 2026

Toda-Krylov adaptive polynomial layer

Implementation & benchmark of arXiv:2608.04850 — Two-dimensional Toda--Arnoldi correspondence: Holomorphic Krylov geometry and counterdiabatic transport

Usefulness7/10

Difficulty5/10

Novelty7/10

Architecture Accuracy Stability Scalability Ssm Graph nn Mlp Inference speedup Linear algebra Dynamical systems Geometry

Source paper: Two-dimensional Toda--Arnoldi correspondence: Holomorphic Krylov geometry and counterdiabatic transport arXiv:2608.04850 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a fixed-order polynomial or recurrent state-space block by an Arnoldi basis built from a learned operator and the current input, and use subdiagonal coefficients as geometry-aware gates over Krylov depth. The gates quantify how much genuinely new direction each operator application contributes, allowing the layer to stop early near Krylov breakdown and suppress redundant or unstable directions.

Formulas

$$\mathsf{K}_n(x)=[x,\mathsf{H}x,\ldots,\mathsf{H}^{n-1}x],\qquad \tau_n=\det\!\left(\mathsf{K}_n(x)^*\mathsf{K}_n(x)\right),\qquad \tau_0=1.$$

$$b_n^2=\frac{\tau_{n+1}\tau_{n-1}}{\tau_n^2},\qquad \tau_n>0\Longleftrightarrow \operatorname{rank}(\mathsf{K}_n)=n.$$

$$\mathsf{H}\mathsf{U}=\mathsf{U}\mathsf{A},\qquad \mathsf{A}=\begin{pmatrix}a_0&h_{0,1}&\cdots&h_{0,D-1}\\ b_1&a_1&\ddots&\vdots\\ \vdots&\ddots&\ddots&h_{D-2,D-1}\\ 0&\cdots&b_{D-1}&a_{D-1}\end{pmatrix}.$$

$$g_n=\frac{b_{n+1}^2}{b_{n+1}^2+\epsilon},\qquad k=\min\{n:b_{n+1}^2<\delta\},\qquad y=\sum_{n=0}^{k-1}g_n u_n f_n(\mathsf{A}).$$

Mathematical statement

For a seed vector x in C^D, define the Krylov matrix K_n(x) = [x, Hx, ..., H^(n-1)x], where H is a learned square operator. Its Gram determinant is tau_n = det(K_n(x)^* K_n(x)), with tau_0 = 1. Arnoldi orthogonalization produces an orthonormal basis u_0, ..., u_(d_K-1) and an upper-Hessenberg matrix A satisfying H U = U A. The subdiagonal entries are residual norms b_(n+1) >= 0. The Toda/Krylov identity is b_n^2 = tau_(n+1) tau_(n-1) / tau_n^2, with indexing shifted depending on whether the seed is numbered zero or one. The paper's Arnoldi matrix has diagonal entries a_n, upper entries h_(m,n), and only nearest-neighbor forward entries b_(n+1). On the real section, tau_n > 0 is equivalent to rank(K_n) = n; when tau_n = 0, the corresponding Krylov geometry is undefined rather than safely assignable a zero coefficient. The ML adaptation uses gates g_n = b_(n+1)^2 / (b_(n+1)^2 + epsilon) to weight successive basis features.

Implementation notes

Integrate this at the linear transformation inside an SSM, graph-NN propagation block, or polynomial MLP layer. Given a minibatch tensor X and a learned operator H, choose a probe vector x from each example or use a batch-shared normalized probe. Build v_0 = x and v_(j+1) = H v_j. Run modified Gram-Schmidt with reorthogonalization: set r_j = v_j - U_(<j)(U_(<j)^* v_j), compute b_(j+1) = norm(r_j)_2, and stop if b_(j+1) < delta. Otherwise append u_j = r_j / (b_(j+1) + epsilon). Compute a_j = u_j^* H u_j and upper entries h_(i,j) = u_i^* H u_j, forming the small Hessenberg matrix A = U^* H U. The block output can be U f(A) U^* x, or a concatenation of u_j features passed through a learned readout. Apply g_j = b_(j+1)^2 / (b_(j+1)^2 + epsilon) before the readout. Compute b_j directly from Arnoldi for numerical robustness; compute tau_j only as a diagnostic and verify b_n^2 = tau_(n+1) tau_(n-1) / tau_n^2 on small examples. Start with a width-64 residual SSM on sequential MNIST or WikiText-103. Compare against a fixed-order degree-8 polynomial block at equal operator multiplications. Measure validation loss, effective Krylov depth, gradient norms, and breakdown frequency. Success means equal or better accuracy with fewer average operator applications, smoother loss, and no increase in exploding-gradient or NaN events.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a reorthogonalized Arnoldi/Toda-Krylov prototype with Gram-determinant diagnostics, geometry gates, adaptive breakdown, and a fixed-order polynomial comparison. The core mechanism manifested: the Toda identity error was 4.9e-14, b² scaled as gamma², the g=0.5 transition occurred at the predicted gamma=0.0010644, and observed Krylov depth matched the number of distinct eigenvalues. However, the adaptive readout did not produce an ML win: it used all 8 matvecs and had substantially higher test MSE than the polynomial baseline.

Agent confidence: 9/10

Baseline: Polynomial degree-8: test MSE 1.146e-18; 8 operator multiplications/example.
Idea: Adaptive Arnoldi: test MSE 5.846e-2; mean 8.0 matvecs/example; depth histogram {8: 120}; identity maximum relative error 4.92e-14; scaling, gate-transition, and rank-depth predictions confirmed.

Limitations: Only a small synthetic vector regression was tested, not sequential MNIST, WikiText, an SSM, graph-NN, GPU execution, FLOP-normalized training, learned end-to-end gates/readout, gradient stability, or noisy/near-breakdown cases. The final reproducibility comparison differed in file-versus-stdout newline formatting, but numerical values are deterministic.

How to run: python3 toda_krylov_mvp.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.04850) Sep 2, 2026

Significantly WORSE than the tuned baseline (+159.9%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0003
Idea mean: 0.0008
Effect (Δ): 0.0005 (+159.9%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±74.1%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the registered built-in dynamics track was tested, with 800 training examples, 300 test examples, 12 epochs, order 4, and fixed epsilon/delta. The implementation evaluated all four Krylov directions and therefore did not demonstrate actual early stopping or inference speedup.

How to run: python3 stage2_krylov_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

The full stage-2 bench was run on the registered dynamics track with 8 paired seeds, tuned baseline sweep, and a parity learning-rate grid. The Toda-Krylov recurrent system confirmed the trained-model gate behavior but had significantly higher test MSE than the GRU baseline, so the proposed transfer did not win.

Baseline: rnn_small GRU, best lr=0.01: mean test MSE 0.0002970023.
Idea: KrylovRNN, best lr=0.003: mean test MSE 0.0007719154; paired delta +0.0004749132; 0/8 wins; p=0.0081.
Mechanism signature: confirmed=true from trained benchmark models; residual-derived gates remained in [0,1], effective depth 4/4.

Benchmark result