Residual-Redundancy Adapter Clustering

1. Integration point: in a multi-task Transformer or encoder, replace a single shared LoRA update $W\leftarrow W+B A$ with $K$ cluster-specific updates $W\leftarrow W+B_kA_k$. The task ID selects cluster $k$, so routing is performed once per example and adds no token-level routing overhead. Keep the base model frozen initially and train the adapters and task heads. 2. Pseudocode: train a shared rank-$r$ adapter for a warm-up period; on a held-out validation buffer, collect residual vectors $e_i=[e_i^{(1)},\ldots,e_i^{(T)}]$ with $e_i^{(t)}=y_i^{(t)}-\hat y_i^{(t)}$; compute the sample covariance; apply shrinkage $\widehat\Sigma\leftarrow(1-\alpha)\widehat\Sigma+\alpha\operatorname{diag}(\widehat\Sigma)$; for each candidate task group $G$, calculate $TC_G=0.5[\sum_{t\in G}\log(\widehat\Sigma_{tt}+\epsilon)-\log\det(\widehat\Sigma_G+\epsilon I)]$; use agglomerative clustering with merge cost $TC_{A\cup B}-TC_A-TC_B$, stopping at a parameter-budget-selected $K$; instantiate one LoRA adapter per group and continue training. Re-estimate every 1--3 epochs, but only change assignments when the score improvement exceeds a hysteresis threshold. 3. The Gaussian total-correlation estimator, covariance shrinkage, and merge score are computed from the paper's mathematics; $K$, rank allocation, shrinkage $\alpha$, reassignment interval, and threshold are empirical hyperparameters. 4. First cheap experiment: use T5-small or a small shared encoder on six mixed classification and regression tasks. Compare one shared rank-8 LoRA, independent task-specific rank-8 LoRAs, random equal-size partitions, gradient-cosine clustering, and residual-TC clustering at equal total adapter parameters. Track mean score, worst-task score, validation residual TC, parameter count, and training memory. The expected signal is lower residual TC and better mean or worst-task validation performance than random and globally shared adapters at the same adapter budget.