Capacity and Redundancy Trade-offs in Multi-Task Learning

What the math gives to ML

The paper provides an information-theoretic way to distinguish useful multi-task sharing from interference caused by insufficient shared capacity. Its chain-rule sandwich relates the sum of task-wise information in a shared representation to joint label information and total-correlation redundancy, making unresolved task dependence a measurable target for architecture design. A practical transfer is residual-aware task clustering for LoRA adapters: estimate dependence from validation residual covariance, group tasks with compatible unresolved errors, and assign separate low-rank modules to groups with weak compatibility. This evaluates coupling after the current representation has made predictions rather than relying only on raw label correlations or initial gradient similarity.