# Shared-support heterogeneous-strength adapters

- ID: 1321
- Canonical URL: https://synthcore.org/idea/1321/shared-support-heterogeneous-strength-adapters
- API JSON: https://synthcore.org/api/idea/1321.json
- API Markdown: https://synthcore.org/api/idea/1321.md
- Verification status: unverified
- Source: [arXiv:2607.23577](https://arxiv.org/abs/2607.23577)
- Category: architecture
- Solves: memory, generalization, scalability
- ML areas: mlp, transformer, moe-routing, pruning
- Math tags: graph-theory, optimization, statistics, linear-algebra
- Ratings: usefulness 5/10; difficulty 5/10; novelty 5/10

## Idea description

Replace a collection of dense task-specific linear layers with a common sparse structural matrix and task-specific edge strengths. All tasks share the same learned connectivity pattern, but retain independent values on active connections, allowing parameter sharing without forcing identical interactions.

## Mathematical statement

The paper represents each population precision matrix as \(\boldsymbol{\Omega}_l=\boldsymbol{\Theta}\odot\boldsymbol{\Gamma}_l\), where \(l\in\{1,\ldots,L\}\) indexes populations or tasks, \(\odot\) is the elementwise Schur-Hadamard product, \(\boldsymbol{\Theta}\) is the shared structural matrix, and \(\boldsymbol{\Gamma}_l\) contains task-specific interaction strengths. The shared support is encoded by \(\overline{\Gamma}_{ij}=\mathbb{I}(|\Theta_{ij}|>0)\), so a zero in \(\Theta\) removes the corresponding interaction for every task. The factorization has scale ambiguity \(\boldsymbol{\Omega}_l=(a\boldsymbol{\Theta})\odot(a^{-1}\boldsymbol{\Gamma}_l)\) for any scalar \(a\neq 0\), so a neural implementation must normalize one factor, for example by constraining the root-mean-square magnitude of each active row of \(\Gamma_l\). Adapt the penalized objective to \(\sum_l \mathcal{L}_l(\Theta\odot\Gamma_l)+\lambda\|\Theta\|_1+\mu\sum_l\|\Gamma_l-\mathbf{1}\|_F^2\), where \(\mathcal{L}_l\) is task \(l\)'s neural loss, \(\lambda\) promotes globally shared sparsity, \(\mu\) keeps task-specific strengths near a baseline, and \(\mathbf{1}\) is an all-ones matrix. The transferable property is global support selection combined with heterogeneous edge amplitudes.

## Key formulas

- $$\boldsymbol{\Omega}_{l}=\boldsymbol{\Theta}\odot\boldsymbol{\Gamma}_{l}=a\boldsymbol{\Theta}\odot\frac{1}{a}\boldsymbol{\Gamma}_{l},\qquad a\neq 0,$$
- $$\overline{\Gamma}_{ij}=\mathbb{I}(|\Theta_{ij}|>0),$$
- $$\min_{\boldsymbol{\Theta},\{\boldsymbol{\Gamma}_l\}_{l=1}^{L}}\;\sum_{l=1}^{L}\mathcal{L}_l(\boldsymbol{\Theta}\odot\boldsymbol{\Gamma}_l)+\lambda\|\boldsymbol{\Theta}\|_1+\mu\sum_{l=1}^{L}\|\boldsymbol{\Gamma}_l-\mathbf{1}\|_F^2,$$
- $$\boldsymbol{\Theta}\leftarrow\operatorname{soft\_threshold}\!\left(\boldsymbol{\Theta}-\eta_\Theta\nabla_\Theta\sum_l\mathcal{L}_l(\boldsymbol{\Theta}\odot\boldsymbol{\Gamma}_l),\;\eta_\Theta\lambda\right).$$

## Implementation notes

(1) Integration point: apply this to the task-conditioned projection in a multi-domain MLP, transformer feed-forward block, or small MoE expert bank. For an input width d and output width h, store one shared structural matrix Theta in R^{h x d} and L task-specific strength matrices Gamma[l] in R^{h x d}. For task l, compute W[l] = Theta * Gamma[l] using elementwise multiplication, then use W[l] in the linear projection. Initially restrict the method to a final classifier, adapter, or one feed-forward block rather than the entire transformer.

(2) Pseudocode:
```text
initialize unconstrained Z with Xavier(d,h)
initialize Gamma[l] = ones(h,d) for l=1..L
for minibatch (x,y,task=l):
    Theta = sign(Z) * relu(abs(Z) - tau)
    W = Theta * Gamma[l]
    loss = task_loss(W @ x, y) + mu * sum((Gamma[l]-1)^2)
    update Gamma[l] using Adam on grad(loss, Gamma[l])
    update Z using Adam on grad(loss + lambda*sum(abs(Theta)), Z)
    every K steps: Gamma[l] = Gamma[l] / (rms(Gamma[l]) + epsilon)
```
Here tau is eta times lambda, and soft-thresholding is applied as sign(z) max(abs(z)-tau,0). Use a straight-through estimator through the threshold or alternate differentiable training with periodic hard pruning.

(3) Computed directly from the mathematical construction: the Hadamard factorization, global L1 support penalty, task-specific Frobenius shrinkage, and scale normalization. Estimated empirically: lambda and mu by validation, the support density, support stability across random seeds, and whether sparse kernels actually improve wall-clock time. After training, compile the nonzero entries of Theta into indexed sparse storage and apply only active connections.

(4) First cheap experiment: train a six-domain MLP on Split-MNIST or a six-domain tabular classification benchmark. Compare independent dense heads, one shared dense head, group-lasso sharing, and the proposed factorization at matched parameter budgets. Sweep target support densities of 25%, 50%, and 75%. Success means lower stored parameters and measured inference latency than independent heads at equal accuracy, with no more than a 1% absolute accuracy loss. Also measure few-shot performance per task; the method should outperform independent heads when each domain has limited data because support is pooled across domains.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.