✗ Failed on benchmark 2026

Finite-Excitation Orthogonal Gradient Memory

Implementation & benchmark of arXiv:2608.22562 — Robust Model Reference Adaptive Control with Combined Adaptation under Finite Excitation Condition

Usefulness7/10

Difficulty5/10

Novelty7/10

Dynamics Stability Speedup Generalization Optimizer Training dynamics Memory Mlp Control theory Dynamical systems Linear algebra Optimization

Source paper: Robust Model Reference Adaptive Control with Combined Adaptation under Finite Excitation Condition arXiv:2608.22562 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

For a neural network with a trainable linear head or low-rank adapter, store feature vectors from recent minibatches and select a finite set that is sufficiently independent. Apply Modified Gram-Schmidt to obtain orthonormalized memory directions, then add residual corrections along these directions so the local parameter-error dynamics have an identity coefficient matrix rather than a poorly conditioned empirical Gramian. The method predicts a sharp transition after the buffer first contains enough independent directions: convergence should become insensitive to the original feature excitation level, up to noise and approximation errors.

Formulas

$$\|\Phi^{-1}\|\leq\gamma_{FE},\qquad \Phi=[\phi(t_1),\ldots,\phi(t_d)],\qquad \sigma_{\min}(\Phi)\geq\gamma_{FE}^{-1}.$$

$$v_i=\phi(t_i)-\sum_{j=1}^{i-1}q_jq_j^\top\phi(t_i),\qquad q_i=\frac{v_i}{\|v_i\|},\qquad Q=[q_1,\ldots,q_d],\quad Q^\top Q=I_d.$$

$$\dot{\hat w}=\gamma\sum_{i=1}^{d}q_i\big(y_i-q_i^\top\hat w\big),\qquad \dot{\tilde w}=-\gamma QQ^\top\tilde w=-\gamma\tilde w,$$

$$V=\frac12\|\tilde w\|^2,\qquad \dot V=-\gamma\|\tilde w\|^2=-2\gamma V,\qquad \|\tilde w(t)\|\leq e^{-\gamma(t-t_0)}\|\tilde w(t_0)\|.$$

Mathematical statement

Let $\phi(t)\in\mathbb{R}^{d}$ be a bounded feature or regressor vector and let $\Phi=[\phi(t_1),\ldots,\phi(t_d)]\in\mathbb{R}^{d\times d}$ be a finite memory matrix. The finite-excitation condition is that at least one window contains $d$ samples with $\|\Phi^{-1}\|\leq\gamma_{FE}$, equivalently $\sigma_{\min}(\Phi)\geq\gamma_{FE}^{-1}$. Modified Gram-Schmidt constructs $q_1,\ldots,q_d$ by $v_i=\phi(t_i)-\sum_{j<i}q_jq_j^\top\phi(t_i)$ and $q_i=v_i/\|v_i\|$, assuming every residual norm is nonzero; $Q=[q_1,\ldots,q_d]$ then satisfies $Q^\top Q=I_d$. For a linear head with parameter error $\tilde w=w-w^\star$, use the memory correction $\dot{\hat w}=\gamma\sum_{i=1}^{d}q_i(y_i-q_i^\top\hat w)$, where $\gamma>0$ is the adaptation gain and $(q_i,y_i)$ are stored transformed samples. In the noiseless realizable case, $\dot{\tilde w}=-\gamma QQ^\top\tilde w=-\gamma\tilde w$, so $V=\frac12\|\tilde w\|^2$ obeys $\dot V=-\gamma\|\tilde w\|^2=-2\gamma V$. With bounded transformed error, convergence is to an explicit noise-dependent neighborhood. Before finite excitation, only the currently spanned subspace contracts; after excitation, all $d$ directions contract at rate $\gamma$, independent of $\sigma_{\min}(\Phi)$.

Implementation notes

1) Integration point: use this as a second-order correction for a trainable linear classification or regression head, LoRA adapter, or final layer of an MLP. Freeze the backbone for the first MVP so the feature dimension is manageable. Let $z_b\in\mathbb{R}^d$ be the detached feature for example $b$, with prediction $\hat y_b=\hat Wz_b$. Maintain a memory of feature-target pairs and combine ordinary Adam with the orthogonal memory correction. 2) Pseudocode: every $K$ steps, normalize candidate features, greedily select $d$ vectors whose Gram-Schmidt residual exceeds $\varepsilon$, calculate $v_i=z_i-\sum_{j<i}q_j(q_j^\top z_i)$, and set $q_i=v_i/(\|v_i\|+10^{-8})$. For each selected direction compute $r_i=y_i-\hat Wq_i$, form $g_{mem}=\sum_i r_iq_i^\top$, and update $\hat W\leftarrow\hat W-\eta(g_{batch}+\lambda g_{mem})$, with signs chosen consistently with the loss gradient. Recompute the basis only when the selected set changes, and reject it unless $\min_i\|v_i\|>\varepsilon$. 3) Computed quantities are the MGS basis, residuals, singular values, and memory gradient; empirical quantities are the excitation threshold, label-noise level, and best $\lambda$. For nonlinear backbones, refresh features periodically and apply the correction only to the adapter. 4) First cheap experiment: train a two-layer MLP with a 10-dimensional linear head on MNIST, comparing Adam, unorthogonalized replay-gradient memory, and MGS memory at equal memory size and compute. Create low-excitation phases by restricting batches to one class, then high-excitation phases using many classes. Measure the smallest singular value of the selected feature matrix and head-error decay after a basis reset. The prediction is a transition when $d$ independent directions become available: afterward, the slope of $\log\|\tilde W\|$ should approach $-\eta\lambda$ and vary by less than 20% across buffers with different $\sigma_{\min}(\Phi)$; before it, unexcited directions should not contract. In discrete time, excessive correction strength should produce a stability boundary approximately $\eta\lambda<2$ for normalized directions.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built and numerically verified finite-excitation orthogonal-gradient memory. The discrete stability boundary occurred at alpha=2, full-direction contraction began exactly when k=d independent directions were available, and MGS achieved the predicted conditioning-independent decay slope. In the controlled ill-conditioned replay test, raw replay became unstable while MGS remained stable, though this is not evidence of a broad neural-network training win.

Agent confidence: 9/10

Baseline: Raw replay on ill-conditioned features: late log-error slope +1.5065; error at step 10 6.96e5. On well-conditioned features: slope -1.6094.
Idea: MGS memory: max |Q^TQ-I| 2.83e-10; stable at alpha=1.99 and divergent at 2.01; full excitation at k=8; late log-error slope -1.6094 for both well- and ill-conditioned features.

Limitations: Tested only deterministic NumPy linear-head dynamics. No MNIST/MLP, Adam integration, minibatch feature refresh, label noise, nonlinear backbone, compute comparison, or generalization evaluation was tested.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.22562) Sep 2, 2026

Ahead of the baseline (−1.0%) but not statistically significant (p=0.0939, wins 6 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 21.8103
Idea mean: 21.5959
Effect (Δ): -0.2143 (−1.0%; negative = idea better)
Wins: 6 / 8 paired seeds
p-value: 0.0939 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±1.3%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched built-in tabular track was tested. The run used a short 8-epoch budget and did not test vision, sequence, or dynamics. The mechanism diagnostic should be corrected and independently rerun before making stronger claims about conditioning transfer.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented finite-excitation MGS memory as an optimizer-side correction to the final linear head of the registered tabular mlp_tiny benchmark. The idea modestly improved test MSE from 21.8103 to 21.5959, but the paired permutation test gave p=0.09385, so it did not meet the required significant-win criterion. The trained-model mechanism signature was also not confirmed: observed Gram diagnostics were inconsistent with the identity prediction.

Baseline: Tuned Adam baseline, lr=0.003, weight_decay=0: mean test MSE 21.810271620750427, std 6.031322024802775, 8 paired seeds.
Idea: MGS memory, lr=0.003, lambda=0.1: mean test MSE 21.595937252044678, std 5.868687330662625, 8 paired seeds; paired delta -0.2143343687057495, six wins, p=0.09385.

Benchmark result