Interference and Retention in Continual Learning

What the math gives to ML

The paper gives a precise quadratic notion of forgetting in the frozen-feature regime: the loss increase on an old task is exactly the old task's feature-covariance energy of the parameter displacement. This reveals why isotropic parameter penalties are inefficient: movement in directions outside the old task's covariance support is harmless, whereas movement along high-variance directions causes large interference. The most direct neural-network transfer is a covariance-aware trust-region optimizer for linear heads, adapters, or LoRA parameters that limits old-task interference without replay or Fisher estimation. It should preserve more plasticity than unconditional gradient projection while maintaining comparable retention.