Finite-Excitation Orthogonal Gradient Memory

Implementation

experiment.py implements modified Gram-Schmidt memory construction and three deterministic checks. results.json contains the complete numerical output.

Quantitative mechanism checks

1. Orthogonalization claim. An 8D feature matrix with singular values geometrically decreasing from 1 to 1e-7 had sigma_min=1.0000000002e-7. MGS retained rank 8 and produced max|Q.T Q-I|=2.83e-10.

2. Discrete stability boundary. For a complete orthonormal memory, the update is e_next=(1-alpha)e. The predicted boundary is alpha=2 (equivalently eta*lambda=2). The observed sweep was stable at alpha=1.99 (norm after 30 steps 0.7397) and unstable at alpha=2.01 (norm 1.3478); observed norms matched |1-alpha|^30 to floating-point precision. The sweep also included alpha 0.25, 0.75, 1.0, 1.25, 1.75, and 2.5.

3. Finite-excitation transition. With dimension 8 and alpha=0.8, a memory spanning k orthogonal directions contracts only those coordinates. At step 15, observed norms for k=0..8 were respectively 1.0000, 0.9354, 0.8660, 0.7906, 0.7071, 0.6124, 0.5000, 0.3536, 3.28e-11, matching the closed-form prediction to numerical precision. Full all-direction contraction first appeared exactly at the predicted transition k=8.

4. Conditioning prediction. With the same dimension and gain, raw replay on a well-conditioned matrix (sigma_min=1) had late log-error slope -1.6094, while MGS had -1.6094. On an ill-conditioned matrix (sigma_min=0.001), raw replay had slope +1.5065 and error at step 10 6.96e5 (unstable under the same gain), while MGS retained slope -1.6094 and error 1.02e-7. Thus MGS removed the dependence on the feature singular spectrum in this controlled linear-head setting, though the comparison uses an intentionally aggressive common step size that exposes raw replay's stability problem.

Reproduction

/home/maxwelhelp/main/bin/python3 experiment.py

All tests use NumPy and fixed seed 2688; no GPU or large model is required.