# Finite-Excitation Orthogonal Gradient Memory

- ID: 2688
- Canonical URL: https://synthcore.org/idea/2688/finite-excitation-orthogonal-gradient-memory
- API JSON: https://synthcore.org/api/idea/2688.json
- API Markdown: https://synthcore.org/api/idea/2688.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.22562](https://arxiv.org/abs/2608.22562)
- Category: dynamics
- Solves: stability, speedup, generalization
- ML areas: optimizer, training-dynamics, memory, mlp
- Math tags: control-theory, dynamical-systems, linear-algebra, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

For a neural network with a trainable linear head or low-rank adapter, store feature vectors from recent minibatches and select a finite set that is sufficiently independent. Apply Modified Gram-Schmidt to obtain orthonormalized memory directions, then add residual corrections along these directions so the local parameter-error dynamics have an identity coefficient matrix rather than a poorly conditioned empirical Gramian. The method predicts a sharp transition after the buffer first contains enough independent directions: convergence should become insensitive to the original feature excitation level, up to noise and approximation errors.

## Mathematical statement

Let \(\phi(t)\in\mathbb{R}^{d}\) be a bounded feature or regressor vector and let \(\Phi=[\phi(t_1),\ldots,\phi(t_d)]\in\mathbb{R}^{d\times d}\) be a finite memory matrix. The finite-excitation condition is that at least one window contains \(d\) samples with \(\|\Phi^{-1}\|\leq\gamma_{FE}\), equivalently \(\sigma_{\min}(\Phi)\geq\gamma_{FE}^{-1}\). Modified Gram-Schmidt constructs \(q_1,\ldots,q_d\) by \(v_i=\phi(t_i)-\sum_{j<i}q_jq_j^\top\phi(t_i)\) and \(q_i=v_i/\|v_i\|\), assuming every residual norm is nonzero; \(Q=[q_1,\ldots,q_d]\) then satisfies \(Q^\top Q=I_d\). For a linear head with parameter error \(\tilde w=w-w^\star\), use the memory correction \(\dot{\hat w}=\gamma\sum_{i=1}^{d}q_i(y_i-q_i^\top\hat w)\), where \(\gamma>0\) is the adaptation gain and \((q_i,y_i)\) are stored transformed samples. In the noiseless realizable case, \(\dot{\tilde w}=-\gamma QQ^\top\tilde w=-\gamma\tilde w\), so \(V=\frac12\|\tilde w\|^2\) obeys \(\dot V=-\gamma\|\tilde w\|^2=-2\gamma V\). With bounded transformed error, convergence is to an explicit noise-dependent neighborhood. Before finite excitation, only the currently spanned subspace contracts; after excitation, all \(d\) directions contract at rate \(\gamma\), independent of \(\sigma_{\min}(\Phi)\).

## Key formulas

- $$\|\Phi^{-1}\|\leq\gamma_{FE},\qquad \Phi=[\phi(t_1),\ldots,\phi(t_d)],\qquad \sigma_{\min}(\Phi)\geq\gamma_{FE}^{-1}.$$
- $$v_i=\phi(t_i)-\sum_{j=1}^{i-1}q_jq_j^\top\phi(t_i),\qquad q_i=\frac{v_i}{\|v_i\|},\qquad Q=[q_1,\ldots,q_d],\quad Q^\top Q=I_d.$$
- $$\dot{\hat w}=\gamma\sum_{i=1}^{d}q_i\big(y_i-q_i^\top\hat w\big),\qquad \dot{\tilde w}=-\gamma QQ^\top\tilde w=-\gamma\tilde w,$$
- $$V=\frac12\|\tilde w\|^2,\qquad \dot V=-\gamma\|\tilde w\|^2=-2\gamma V,\qquad \|\tilde w(t)\|\leq e^{-\gamma(t-t_0)}\|\tilde w(t_0)\|.$$

## Implementation notes

1) Integration point: use this as a second-order correction for a trainable linear classification or regression head, LoRA adapter, or final layer of an MLP. Freeze the backbone for the first MVP so the feature dimension is manageable. Let \(z_b\in\mathbb{R}^d\) be the detached feature for example \(b\), with prediction \(\hat y_b=\hat Wz_b\). Maintain a memory of feature-target pairs and combine ordinary Adam with the orthogonal memory correction. 2) Pseudocode: every \(K\) steps, normalize candidate features, greedily select \(d\) vectors whose Gram-Schmidt residual exceeds \(\varepsilon\), calculate \(v_i=z_i-\sum_{j<i}q_j(q_j^\top z_i)\), and set \(q_i=v_i/(\|v_i\|+10^{-8})\). For each selected direction compute \(r_i=y_i-\hat Wq_i\), form \(g_{mem}=\sum_i r_iq_i^\top\), and update \(\hat W\leftarrow\hat W-\eta(g_{batch}+\lambda g_{mem})\), with signs chosen consistently with the loss gradient. Recompute the basis only when the selected set changes, and reject it unless \(\min_i\|v_i\|>\varepsilon\). 3) Computed quantities are the MGS basis, residuals, singular values, and memory gradient; empirical quantities are the excitation threshold, label-noise level, and best \(\lambda\). For nonlinear backbones, refresh features periodically and apply the correction only to the adapter. 4) First cheap experiment: train a two-layer MLP with a 10-dimensional linear head on MNIST, comparing Adam, unorthogonalized replay-gradient memory, and MGS memory at equal memory size and compute. Create low-excitation phases by restricting batches to one class, then high-excitation phases using many classes. Measure the smallest singular value of the selected feature matrix and head-error decay after a basis reset. The prediction is a transition when \(d\) independent directions become available: afterward, the slope of \(\log\|\tilde W\|\) should approach \(-\eta\lambda\) and vary by less than 20% across buffers with different \(\sigma_{\min}(\Phi)\); before it, unexcited directions should not contract. In discrete time, excessive correction strength should produce a stability boundary approximately \(\eta\lambda<2\) for normalized directions.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built and numerically verified finite-excitation orthogonal-gradient memory. The discrete stability boundary occurred at alpha=2, full-direction contraction began exactly when k=d independent directions were available, and MGS achieved the predicted conditioning-independent decay slope. In the controlled ill-conditioned replay test, raw replay became unstable while MGS remained stable, though this is not evidence of a broad neural-network training win.

### Mechanism check

- Verdict: Built and numerically verified finite-excitation orthogonal-gradient memory. The discrete stability boundary occurred at alpha=2, full-direction contraction began exactly when k=d independent directions were available, and MGS achieved the predicted conditioning-independent decay slope. In the controlled ill-conditioned replay test, raw replay became unstable while MGS remained stable, though this is not evidence of a broad neural-network training win.
- Confidence: 9/10
- Limitations: Tested only deterministic NumPy linear-head dynamics. No MNIST/MLP, Adam integration, minibatch feature refresh, label noise, nonlinear backbone, compute comparison, or generalization evaluation was tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.09385
- Paired wins: 6/8
- Benchmark verdict: no measurable effect

## Artifacts

- [REPORT.md](https://synthcore.org/code/1038/REPORT.md)
- [bench_experiment.py](https://synthcore.org/code/1038/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/1038/bench_report.json)
- [experiment.py](https://synthcore.org/code/1038/experiment.py)
- [report.md](https://synthcore.org/code/1038/report.md)
- [report_bench_2026-09-01T190929.md](https://synthcore.org/code/1038/report_bench_2026-09-01T190929.md)
- [results.json](https://synthcore.org/code/1038/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1038)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.