{
 "artifacts": [
  {
   "name": "REPORT.md",
   "url": "https://synthcore.org/code/1038/REPORT.md"
  },
  {
   "name": "bench_experiment.py",
   "url": "https://synthcore.org/code/1038/bench_experiment.py"
  },
  {
   "name": "bench_report.json",
   "url": "https://synthcore.org/code/1038/bench_report.json"
  },
  {
   "name": "experiment.py",
   "url": "https://synthcore.org/code/1038/experiment.py"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/1038/report.md"
  },
  {
   "name": "report_bench_2026-09-01T190929.md",
   "url": "https://synthcore.org/code/1038/report_bench_2026-09-01T190929.md"
  },
  {
   "name": "results.json",
   "url": "https://synthcore.org/code/1038/results.json"
  }
 ],
 "category": "dynamics",
 "description": "For a neural network with a trainable linear head or low-rank adapter, store feature vectors from recent minibatches and select a finite set that is sufficiently independent. Apply Modified Gram-Schmidt to obtain orthonormalized memory directions, then add residual corrections along these directions so the local parameter-error dynamics have an identity coefficient matrix rather than a poorly conditioned empirical Gramian. The method predicts a sharp transition after the buffer first contains enough independent directions: convergence should become insensitive to the original feature excitation level, up to noise and approximation errors.",
 "download_zip": "https://synthcore.org/download/1038",
 "formulas_latex": [
  "$$\\|\\Phi^{-1}\\|\\leq\\gamma_{FE},\\qquad \\Phi=[\\phi(t_1),\\ldots,\\phi(t_d)],\\qquad \\sigma_{\\min}(\\Phi)\\geq\\gamma_{FE}^{-1}.$$",
  "$$v_i=\\phi(t_i)-\\sum_{j=1}^{i-1}q_jq_j^\\top\\phi(t_i),\\qquad q_i=\\frac{v_i}{\\|v_i\\|},\\qquad Q=[q_1,\\ldots,q_d],\\quad Q^\\top Q=I_d.$$",
  "$$\\dot{\\hat w}=\\gamma\\sum_{i=1}^{d}q_i\\big(y_i-q_i^\\top\\hat w\\big),\\qquad \\dot{\\tilde w}=-\\gamma QQ^\\top\\tilde w=-\\gamma\\tilde w,$$",
  "$$V=\\frac12\\|\\tilde w\\|^2,\\qquad \\dot V=-\\gamma\\|\\tilde w\\|^2=-2\\gamma V,\\qquad \\|\\tilde w(t)\\|\\leq e^{-\\gamma(t-t_0)}\\|\\tilde w(t_0)\\|.$$"
 ],
 "id": 2688,
 "implementation": "1) Integration point: use this as a second-order correction for a trainable linear classification or regression head, LoRA adapter, or final layer of an MLP. Freeze the backbone for the first MVP so the feature dimension is manageable. Let \\(z_b\\in\\mathbb{R}^d\\) be the detached feature for example \\(b\\), with prediction \\(\\hat y_b=\\hat Wz_b\\). Maintain a memory of feature-target pairs and combine ordinary Adam with the orthogonal memory correction. 2) Pseudocode: every \\(K\\) steps, normalize candidate features, greedily select \\(d\\) vectors whose Gram-Schmidt residual exceeds \\(\\varepsilon\\), calculate \\(v_i=z_i-\\sum_{j\u003ci}q_j(q_j^\\top z_i)\\), and set \\(q_i=v_i/(\\|v_i\\|+10^{-8})\\). For each selected direction compute \\(r_i=y_i-\\hat Wq_i\\), form \\(g_{mem}=\\sum_i r_iq_i^\\top\\), and update \\(\\hat W\\leftarrow\\hat W-\\eta(g_{batch}+\\lambda g_{mem})\\), with signs chosen consistently with the loss gradient. Recompute the basis only when the selected set changes, and reject it unless \\(\\min_i\\|v_i\\|\u003e\\varepsilon\\). 3) Computed quantities are the MGS basis, residuals, singular values, and memory gradient; empirical quantities are the excitation threshold, label-noise level, and best \\(\\lambda\\). For nonlinear backbones, refresh features periodically and apply the correction only to the adapter. 4) First cheap experiment: train a two-layer MLP with a 10-dimensional linear head on MNIST, comparing Adam, unorthogonalized replay-gradient memory, and MGS memory at equal memory size and compute. Create low-excitation phases by restricting batches to one class, then high-excitation phases using many classes. Measure the smallest singular value of the selected feature matrix and head-error decay after a basis reset. The prediction is a transition when \\(d\\) independent directions become available: afterward, the slope of \\(\\log\\|\\tilde W\\|\\) should approach \\(-\\eta\\lambda\\) and vary by less than 20% across buffers with different \\(\\sigma_{\\min}(\\Phi)\\); before it, unexcited directions should not contract. In discrete time, excessive correction strength should produce a stability boundary approximately \\(\\eta\\lambda\u003c2\\) for normalized directions.",
 "math_summary": "Let \\(\\phi(t)\\in\\mathbb{R}^{d}\\) be a bounded feature or regressor vector and let \\(\\Phi=[\\phi(t_1),\\ldots,\\phi(t_d)]\\in\\mathbb{R}^{d\\times d}\\) be a finite memory matrix. The finite-excitation condition is that at least one window contains \\(d\\) samples with \\(\\|\\Phi^{-1}\\|\\leq\\gamma_{FE}\\), equivalently \\(\\sigma_{\\min}(\\Phi)\\geq\\gamma_{FE}^{-1}\\). Modified Gram-Schmidt constructs \\(q_1,\\ldots,q_d\\) by \\(v_i=\\phi(t_i)-\\sum_{j\u003ci}q_jq_j^\\top\\phi(t_i)\\) and \\(q_i=v_i/\\|v_i\\|\\), assuming every residual norm is nonzero; \\(Q=[q_1,\\ldots,q_d]\\) then satisfies \\(Q^\\top Q=I_d\\). For a linear head with parameter error \\(\\tilde w=w-w^\\star\\), use the memory correction \\(\\dot{\\hat w}=\\gamma\\sum_{i=1}^{d}q_i(y_i-q_i^\\top\\hat w)\\), where \\(\\gamma\u003e0\\) is the adaptation gain and \\((q_i,y_i)\\) are stored transformed samples. In the noiseless realizable case, \\(\\dot{\\tilde w}=-\\gamma QQ^\\top\\tilde w=-\\gamma\\tilde w\\), so \\(V=\\frac12\\|\\tilde w\\|^2\\) obeys \\(\\dot V=-\\gamma\\|\\tilde w\\|^2=-2\\gamma V\\). With bounded transformed error, convergence is to an explicit noise-dependent neighborhood. Before finite excitation, only the currently spanned subspace contracts; after excitation, all \\(d\\) directions contract at rate \\(\\gamma\\), independent of \\(\\sigma_{\\min}(\\Phi)\\).",
 "math_tags": [
  "control-theory",
  "dynamical-systems",
  "linear-algebra",
  "optimization"
 ],
 "ml_areas": [
  "optimizer",
  "training-dynamics",
  "memory",
  "mlp"
 ],
 "paper": {
  "arxiv_id": "2608.22562",
  "arxiv_url": "https://arxiv.org/abs/2608.22562",
  "summary_what_math_gives_to_ml": "The paper offers a constructive finite-excitation memory mechanism: once a finite set of regressors is sufficiently independent, Modified Gram-Schmidt constructs an orthogonal memory matrix whose coefficient in the parameter-error dynamics is the identity. This removes the usual dependence of adaptation speed on the regressor Gramian's condition number or excitation magnitude, while retaining an explicit ultimate bound under bounded uncertainty. The most direct neural-network transfer is an orthogonalized gradient-memory optimizer for a linear output head or low-rank adapter, where a finite buffer of feature vectors is compressed into a well-conditioned basis and used to isotropically correct parameter errors.",
  "title": "Robust Model Reference Adaptive Control with Combined Adaptation under Finite Excitation Condition",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 7,
  "usefulness": 7
 },
 "solves": [
  "stability",
  "speedup",
  "generalization"
 ],
 "title": "Finite-Excitation Orthogonal Gradient Memory",
 "url": "https://synthcore.org/idea/2688/finite-excitation-orthogonal-gradient-memory",
 "verification": {
  "benchmark": {
   "track": "tabular",
   "model": "mlp_tiny",
   "n_seeds": 8,
   "metric_direction": "lower is better",
   "comparison": {
    "baseline_mean": 21.810271620750427,
    "idea_mean": 21.595937252044678,
    "delta_mean": -0.2143343687057495,
    "per_seed_diffs": [
     -0.03144645690917969,
     -0.18142414093017578,
     -0.9294929504394531,
     -0.33823204040527344,
     -0.36135101318359375,
     -0.0377349853515625,
     0.10631561279296875,
     0.05869102478027344
    ],
    "idea_wins": 6,
    "n_pairs": 8,
    "p_value": 0.09385,
    "mde": 0.2811647999151475,
    "mde_rel_pct": 1.2891393780151073,
    "verdict": "no measurable effect",
    "system_worked": false
   },
   "baseline": {
    "best_cfg": {
     "lr": 0.003,
     "weight_decay": 0
    },
    "sweep": [
     {
      "cfg": {
       "lr": 0.001,
       "weight_decay": 0
      },
      "mean": 188.35083389282227
     },
     {
      "cfg": {
       "lr": 0.003,
       "weight_decay": 0
      },
      "mean": 19.947585344314575
     },
     {
      "cfg": {
       "lr": 0.006,
       "weight_decay": 0
      },
      "mean": 22.523260593414307
     }
    ],
    "full": {
     "mean": 21.810271620750427,
     "std": 6.031322024802775,
     "per_seed": [
      17.455322265625,
      15.50521183013916,
      29.37735366821289,
      17.45245361328125,
      33.27134323120117,
      16.96589469909668,
      22.887645721435547,
      21.56694793701172
     ],
     "n": 8
    }
   },
   "idea": {
    "mean": 21.595937252044678,
    "std": 5.868687330662625,
    "per_seed": [
     17.42387580871582,
     15.323787689208984,
     28.447860717773438,
     17.114221572875977,
     32.90999221801758,
     16.928159713745117,
     22.993961334228516,
     21.625638961791992
    ],
    "n": 8
   },
   "mechanism_signature": {
    "confirmed": false,
    "idea_configs": [
     {
      "cfg": {
       "lambda": 0.02,
       "lr": 0.003
      },
      "result": {
       "mean": 21.754212260246277,
       "n": 8,
       "per_seed": [
        17.445890426635742,
        15.479540824890137,
        29.154930114746094,
        17.36090087890625,
        33.19727325439453,
        16.9453067779541,
        22.87087631225586,
        21.5789794921875
       ],
       "std": 5.9932442203746445
      }
     },
     {
      "cfg": {
       "lambda": 0.05,
       "lr": 0.003
      },
      "result": {
       "mean": 21.67228603363037,
       "n": 8,
       "per_seed": [
        17.41834259033203,
        15.36572265625,
        28.8680419921875,
        17.256317138671875,
        33.06776428222656,
        16.93799591064453,
        22.85132598876953,
        21.612777709960938
       ],
       "std": 5.945718685399587
      }
     },
     {
      "cfg": {
       "lambda": 0.1,
       "lr": 0.003
      },
      "result": {
       "mean": 21.595937252044678,
       "n": 8,
       "per_seed": [
        17.42387580871582,
        15.323787689208984,
        28.447860717773438,
        17.114221572875977,
        32.90999221801758,
        16.928159713745117,
        22.993961334228516,
        21.625638961791992
       ],
       "std": 5.868687330662625
      }
     }
    ],
    "prediction": {
     "confirmed": false,
     "feature_dim": 64,
     "observed_QtQ_error_max": 1,
     "observed_rank_last": 64,
     "predicted_identity_max_eigen_mean": 1.8750072717666626,
     "prediction": "MGS memory predicts identity Gramian after d independent directions"
    }
   },
   "custom_track": null
  },
  "mechanism_confirmed": false,
  "peer_reviewed": false,
  "practical_verdict": "no_effect",
  "stage1_mechanism_check": {
   "worked": true,
   "confidence": 9,
   "verdict": "Built and numerically verified finite-excitation orthogonal-gradient memory. The discrete stability boundary occurred at alpha=2, full-direction contraction began exactly when k=d independent directions were available, and MGS achieved the predicted conditioning-independent decay slope. In the controlled ill-conditioned replay test, raw replay became unstable while MGS remained stable, though this is not evidence of a broad neural-network training win.",
   "metrics": {
    "baseline": "Raw replay on ill-conditioned features: late log-error slope +1.5065; error at step 10 6.96e5. On well-conditioned features: slope -1.6094.",
    "idea": "MGS memory: max |Q^TQ-I| 2.83e-10; stable at alpha=1.99 and divergent at 2.01; full excitation at k=8; late log-error slope -1.6094 for both well- and ill-conditioned features."
   },
   "how_to_run": "python3 experiment.py",
   "files": [
    "experiment.py",
    "results.json",
    "REPORT.md"
   ],
   "limitations": "Tested only deterministic NumPy linear-head dynamics. No MNIST/MLP, Adam integration, minibatch feature refresh, label noise, nonlinear backbone, compute comparison, or generalization evaluation was tested."
  },
  "status": "failed_benchmark",
  "status_label": "Failed on benchmark",
  "updated_at": "2026-09-01T19:09:29",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}