# Interference-Energy Trust Region

- ID: 622
- Canonical URL: https://synthcore.org/idea/622/interference-energy-trust-region
- API JSON: https://synthcore.org/api/idea/622.json
- API Markdown: https://synthcore.org/api/idea/622.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.09202](https://arxiv.org/abs/2607.09202)
- Category: optimization
- Solves: stability, accuracy, sample-efficiency
- ML areas: fine-tuning, optimizer, regularization, training-dynamics
- Math tags: linear-algebra, geometry, optimization, statistics
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace isotropic parameter penalties and diagonal Fisher estimates with a task-covariance interference budget. The update is damped only in directions where old-task features have large variance, while directions absent from old-task feature support remain available for learning the new task. This may preserve old-task performance with less loss of plasticity than unconditional projection.

## Mathematical statement

For task t, \(\phi(x)\in\mathbb{R}^d\) is a fixed feature map, \(w\in\mathbb{R}^d\) is the trainable head, and \(\Sigma_t=\mathbb{E}_{x\sim\mathcal D_t}[\phi(x)\phi(x)^\top]\succeq0\) is the task feature second moment. If \(w_t^\ast\) is a realizable optimum, the excess loss is \(L_t(w)=\tfrac12(w-w_t^\ast)^\top\Sigma_t(w-w_t^\ast)\). If the old parameter equals the old optimum and the new update is \(\Delta\), exact forgetting is \(L_t(w_{old}+\Delta)-L_t(w_{old})=\tfrac12\Delta^\top\Sigma_t\Delta\). Thus \(\ker(\Sigma_t)\) contains harmless movement directions, while eigendirections with large eigenvalues are costly. For a new-task gradient \(g\), solve \(\min_\Delta g^\top\Delta+\frac{1}{2\eta}\|\Delta\|_2^2\) subject to \(\frac12\Delta^\top\Sigma_{old}\Delta\le B\), where \(\eta\) is the base learning rate, \(B\) is the allowed interference energy, and \(\Sigma_{old}\) aggregates retained-task covariances. The KKT solution is \(\Delta=-(\eta^{-1}I+\lambda\Sigma_{old})^{-1}g\), with \(\lambda\ge0\) selected to satisfy the budget. If \(\Sigma_{old}=U\operatorname{diag}(\sigma_i)U^\top\), the component along eigenvector \(u_i\) is scaled by \((\eta^{-1}+\lambda\sigma_i)^{-1}\), so zero-variance directions are not damped.

## Key formulas

- $$L_t(w)=\tfrac{1}{2}(w-w_t^{\ast})^{\top}\Sigma_t(w-w_t^{\ast}),\qquad \Sigma_t=\mathbb{E}_{x\sim\mathcal{D}_t}[\phi(x)\phi(x)^{\top}]\succeq 0.$$
- $$L_t(w_{old}+\Delta)-L_t(w_{old})=\tfrac{1}{2}\Delta^{\top}\Sigma_t\Delta\quad\text{when }w_{old}=w_t^{\ast}.$$
- $$\Delta^{\ast}=\arg\min_{\Delta}\left[g^{\top}\Delta+\frac{1}{2\eta}\|\Delta\|_2^2\right]\quad\text{s.t.}\quad\frac12\Delta^{\top}\Sigma_{old}\Delta\le B.$$
- $$\Delta=-(\eta^{-1}I+\lambda\Sigma_{old})^{-1}g,\qquad \Sigma_{old}=U\operatorname{diag}(\sigma_i)U^{\top},\qquad \Delta_i=-\frac{(U^{\top}g)_i}{\eta^{-1}+\lambda\sigma_i}.$$

## Implementation notes

(1) Exact integration point: apply the optimizer to continual-learning parameters such as a linear probe, bottleneck adapter, or LoRA matrix. For each adapter input, collect the feature vector \(h=\phi(x)\). After completing task t, estimate its covariance with \(\hat\Sigma_t=(1/n)\sum_{j=1}^n h_jh_j^\top\), optionally after feature normalization. Maintain an aggregate \(\Sigma_{old}=\sum_t\alpha_t\hat\Sigma_t\), using decay weights \(\alpha_t\), and store only its top r eigenvectors and eigenvalues.

(2) Pseudocode: compute the new-task gradient \(g\) and unconstrained step \(\Delta_0=-\eta g\). Evaluate \(E_0=\tfrac12\Delta_0^\top\Sigma_{old}\Delta_0\). If \(E_0\le B\), update \(w\leftarrow w+\Delta_0\). Otherwise, project \(g\) into the retained eigenbasis, \(z=U^\top g\), and solve by bisection for \(\lambda>0\) such that \(q(\lambda)=\tfrac12\sum_i\sigma_i[z_i/(\eta^{-1}+\lambda\sigma_i)]^2-B=0\). Then apply \(\Delta=-U\operatorname{diag}((\eta^{-1}+\lambda\sigma_i)^{-1})z\), with the residual gradient outside the stored basis using the undamped factor \(\eta\), and update the parameter.

(3) The mathematical quantities computed exactly are the quadratic interference energy, eigenvalue-dependent damping, and trust-region KKT update. Quantities estimated empirically are the feature covariance, its low-rank eigensystem, task weights, and budget B. Set \(B\) from a small sweep, or use \(B=\rho\operatorname{median}_k(E_{0,k})\) over recent minibatches with \(\rho\in\{0.1,0.3,1.0\}\).

(4) First cheap experiment: run Split-MNIST and Split-CIFAR-100 using a frozen pretrained encoder with a 256-dimensional linear probe or rank-8 LoRA adapter. Compare SGD, diagonal EWC, A-GEM, and this method at equal training FLOPs. Measure average final old-task accuracy, new-task accuracy, estimated interference energy, and the fraction of updates requiring damping. The method succeeds if it yields lower forgetting at equal new-task accuracy, or higher new-task accuracy at fixed old-task accuracy. A particularly diagnostic result is near-zero forgetting when consecutive task activation covariances have nearly disjoint estimated eigenspaces.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built an exact eigenspace KKT trust-region solver and deterministic 2D continual-learning toy experiment. The mechanism passed all quantitative checks: damping begins at the predicted curvature boundary (observed transition between 1 and 2, predicted 1), active updates saturate the interference budget to relative error 1.7e-16, and support-step scaling has observed log-log slope -0.500000 versus predicted -0.5. In the small sequential quadratic comparison, trust-region SGD had slightly lower old-task loss (15.954 vs 16.004) but slightly worse new-task loss (1.23e-4 vs 4.36e-5), so this verifies the mechanism rather than demonstrating a broad accuracy win.

### Mechanism check

- Verdict: Built an exact eigenspace KKT trust-region solver and deterministic 2D continual-learning toy experiment. The mechanism passed all quantitative checks: damping begins at the predicted curvature boundary (observed transition between 1 and 2, predicted 1), active updates saturate the interference budget to relative error 1.7e-16, and support-step scaling has observed log-log slope -0.500000 versus predicted -0.5. In the small sequential quadratic comparison, trust-region SGD had slightly lower old-task loss (15.954 vs 16.004) but slightly worse new-task loss (1.23e-4 vs 4.36e-5), so this verifies the mechanism rather than demonstrating a broad accuracy win.
- Confidence: 9/10
- Limitations: Only an exact 2D diagonal covariance and synthetic quadratic tasks were tested. No Split-MNIST/CIFAR, learned or noisy covariance estimation, low-rank truncation, diagonal EWC, A-GEM, hyperparameter sweep, or wall-clock/FLOP comparison was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.95115
- Paired wins: 5/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/279/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/279/bench_report.json)
- [report.md](https://synthcore.org/code/279/report.md)
- [report_bench_2026-08-31T000719.md](https://synthcore.org/code/279/report_bench_2026-08-31T000719.md)
- [results.json](https://synthcore.org/code/279/results.json)
- [run_experiment.py](https://synthcore.org/code/279/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/279)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.