✗ Failed on benchmark 2026

Interference-Energy Trust Region

Implementation & benchmark of arXiv:2607.09202 — Interference and Retention in Continual Learning

Usefulness7/10

Difficulty5/10

Novelty6/10

Optimization Stability Accuracy Sample efficiency Fine tuning Optimizer Regularization Training dynamics Linear algebra Geometry Optimization Statistics

Source paper: Interference and Retention in Continual Learning arXiv:2607.09202 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace isotropic parameter penalties and diagonal Fisher estimates with a task-covariance interference budget. The update is damped only in directions where old-task features have large variance, while directions absent from old-task feature support remain available for learning the new task. This may preserve old-task performance with less loss of plasticity than unconditional projection.

Formulas

$$L_t(w)=\tfrac{1}{2}(w-w_t^{\ast})^{\top}\Sigma_t(w-w_t^{\ast}),\qquad \Sigma_t=\mathbb{E}_{x\sim\mathcal{D}_t}[\phi(x)\phi(x)^{\top}]\succeq 0.$$

$$L_t(w_{old}+\Delta)-L_t(w_{old})=\tfrac{1}{2}\Delta^{\top}\Sigma_t\Delta\quad\text{when }w_{old}=w_t^{\ast}.$$

$$\Delta^{\ast}=\arg\min_{\Delta}\left[g^{\top}\Delta+\frac{1}{2\eta}\|\Delta\|_2^2\right]\quad\text{s.t.}\quad\frac12\Delta^{\top}\Sigma_{old}\Delta\le B.$$

$$\Delta=-(\eta^{-1}I+\lambda\Sigma_{old})^{-1}g,\qquad \Sigma_{old}=U\operatorname{diag}(\sigma_i)U^{\top},\qquad \Delta_i=-\frac{(U^{\top}g)_i}{\eta^{-1}+\lambda\sigma_i}.$$

Mathematical statement

For task t, $\phi(x)\in\mathbb{R}^d$ is a fixed feature map, $w\in\mathbb{R}^d$ is the trainable head, and $\Sigma_t=\mathbb{E}_{x\sim\mathcal D_t}[\phi(x)\phi(x)^\top]\succeq0$ is the task feature second moment. If $w_t^\ast$ is a realizable optimum, the excess loss is $L_t(w)=\tfrac12(w-w_t^\ast)^\top\Sigma_t(w-w_t^\ast)$. If the old parameter equals the old optimum and the new update is $\Delta$, exact forgetting is $L_t(w_{old}+\Delta)-L_t(w_{old})=\tfrac12\Delta^\top\Sigma_t\Delta$. Thus $\ker(\Sigma_t)$ contains harmless movement directions, while eigendirections with large eigenvalues are costly. For a new-task gradient $g$, solve $\min_\Delta g^\top\Delta+\frac{1}{2\eta}\|\Delta\|_2^2$ subject to $\frac12\Delta^\top\Sigma_{old}\Delta\le B$, where $\eta$ is the base learning rate, $B$ is the allowed interference energy, and $\Sigma_{old}$ aggregates retained-task covariances. The KKT solution is $\Delta=-(\eta^{-1}I+\lambda\Sigma_{old})^{-1}g$, with $\lambda\ge0$ selected to satisfy the budget. If $\Sigma_{old}=U\operatorname{diag}(\sigma_i)U^\top$, the component along eigenvector $u_i$ is scaled by $(\eta^{-1}+\lambda\sigma_i)^{-1}$, so zero-variance directions are not damped.

Implementation notes

(1) Exact integration point: apply the optimizer to continual-learning parameters such as a linear probe, bottleneck adapter, or LoRA matrix. For each adapter input, collect the feature vector $h=\phi(x)$. After completing task t, estimate its covariance with $\hat\Sigma_t=(1/n)\sum_{j=1}^n h_jh_j^\top$, optionally after feature normalization. Maintain an aggregate $\Sigma_{old}=\sum_t\alpha_t\hat\Sigma_t$, using decay weights $\alpha_t$, and store only its top r eigenvectors and eigenvalues.

(2) Pseudocode: compute the new-task gradient $g$ and unconstrained step $\Delta_0=-\eta g$. Evaluate $E_0=\tfrac12\Delta_0^\top\Sigma_{old}\Delta_0$. If $E_0\le B$, update $w\leftarrow w+\Delta_0$. Otherwise, project $g$ into the retained eigenbasis, $z=U^\top g$, and solve by bisection for $\lambda>0$ such that $q(\lambda)=\tfrac12\sum_i\sigma_i[z_i/(\eta^{-1}+\lambda\sigma_i)]^2-B=0$. Then apply $\Delta=-U\operatorname{diag}((\eta^{-1}+\lambda\sigma_i)^{-1})z$, with the residual gradient outside the stored basis using the undamped factor $\eta$, and update the parameter.

(3) The mathematical quantities computed exactly are the quadratic interference energy, eigenvalue-dependent damping, and trust-region KKT update. Quantities estimated empirically are the feature covariance, its low-rank eigensystem, task weights, and budget B. Set $B$ from a small sweep, or use $B=\rho\operatorname{median}_k(E_{0,k})$ over recent minibatches with $\rho\in\{0.1,0.3,1.0\}$.

(4) First cheap experiment: run Split-MNIST and Split-CIFAR-100 using a frozen pretrained encoder with a 256-dimensional linear probe or rank-8 LoRA adapter. Compare SGD, diagonal EWC, A-GEM, and this method at equal training FLOPs. Measure average final old-task accuracy, new-task accuracy, estimated interference energy, and the fraction of updates requiring damping. The method succeeds if it yields lower forgetting at equal new-task accuracy, or higher new-task accuracy at fixed old-task accuracy. A particularly diagnostic result is near-zero forgetting when consecutive task activation covariances have nearly disjoint estimated eigenspaces.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built an exact eigenspace KKT trust-region solver and deterministic 2D continual-learning toy experiment. The mechanism passed all quantitative checks: damping begins at the predicted curvature boundary (observed transition between 1 and 2, predicted 1), active updates saturate the interference budget to relative error 1.7e-16, and support-step scaling has observed log-log slope -0.500000 versus predicted -0.5. In the small sequential quadratic comparison, trust-region SGD had slightly lower old-task loss (15.954 vs 16.004) but slightly worse new-task loss (1.23e-4 vs 4.36e-5), so this verifies the mechanism rather than demonstrating a broad accuracy win.

Agent confidence: 9/10

Baseline: SGD: old loss 16.0043, new loss 4.36e-05, final w=(-1.0003, 2.0093)
Idea: Interference trust region: old loss 15.9544, new loss 1.23e-04, final w=(-0.9971, 2.0154); budget relative error 1.73e-16; inverse-square-root slope -0.500000

Limitations: Only an exact 2D diagonal covariance and synthetic quadratic tasks were tested. No Split-MNIST/CIFAR, learned or noisy covariance estimation, low-rank truncation, diagonal EWC, A-GEM, hyperparameter sweep, or wall-clock/FLOP comparison was performed.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.09202) Sep 2, 2026

Ahead of the baseline (−0.0%) but not statistically significant (p=0.9512, wins 5 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.027
Idea mean: 7.0265
Effect (Δ): -0.0005 (−0.0%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.9512 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.2%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 6 configs

Limitations:

Only the structurally matched built-in tabular track was tested; vision, sequence, and dynamics were not applicable to this optimizer/regularizer idea. The implementation used a single frozen penultimate-feature second moment collected before training, a scalar-output head, and a fixed budget based on the first minibatch rather than continual-task covariance aggregation. No low-rank truncation, diagonal EWC, A-GEM, multi-task sequential forgetting evaluation, or wall-clock/FLOP comparison was run.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented the Interference-Energy Trust Region as a local optimization intervention for the structurally matched tabular Friedman#1 benchmark, using the same mlp_tiny architecture and paired seeds as SGD. The idea had test MSE 7.02650 versus baseline 7.02696, but the paired permutation p-value was 0.95115, so this is no measurable effect rather than a significant win. The measured mechanism signature showed damping on 8.04% of updates, but average observed energy was only 0.230 of the fixed budget, so the quantitative signature was not confirmed.

Baseline: Full 8-seed tuned SGD: mean MSE 7.0269587, std 0.2907099; best config lr=0.003, weight_decay=0.0001.
Idea: Full 8-seed trust-region MLP: mean MSE 7.0265045, std 0.2935835; best idea config lr=0.003, rho=0.1.
Mechanism signature: map[budget_ratio_observed_to_budget:0.2303994656814497 confirmed:false observed_damped_fraction:0.08040364583333333 observed_update_energy_mean:5.574681954679609e-05 predicted_active_fraction:0.08040364583333333 predicted_unconstrained_energy_mean:0.00024812968714002504]
Paired delta: -0.00045425 MSE (idea minus baseline; lower is better), 5/8 idea wins, permutation p=0.95115.

Benchmark result