# Shared-Private Matrix-Weighted Expert Layers

- ID: 546
- Canonical URL: https://synthcore.org/idea/546/shared-private-matrix-weighted-expert-layers
- API JSON: https://synthcore.org/api/idea/546.json
- API Markdown: https://synthcore.org/api/idea/546.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.06970](https://arxiv.org/abs/2607.06970)
- Category: architecture
- Solves: generalization, scalability, stability
- ML areas: moe-routing, transformer, graph-nn
- Math tags: graph-theory, linear-algebra, geometry, regularization
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Build a multi-expert or multi-task layer whose feature channels are divided into a globally shared subspace and expert-private subspaces. Matrix-weighted message passing couples experts only through selected feature directions, while the nullspace preserves specialization; the graph-cut condition provides a concrete test that the shared channels can propagate across all experts rather than becoming disconnected islands.

## Mathematical statement

Let z_i\in\mathbb{R}^d denote the representation or parameter vector of expert or task i, and let \mathbb{V}=\operatorname{span}(U) be the r-dimensional shared feature subspace with orthogonal projector P_{\mathbb{V}}=UU^T. A matrix-weighted expert graph uses symmetric positive-semidefinite A_{ij} to penalize only differences visible through A_{ij}. The paper's necessary-and-sufficient condition is $$\mathbb{V}\perp\left(\bigcap_{(i,j)\in\mathcal{E}(S,\bar S)}\operatorname{null}(A_{ij})\right),\ \forall S\subseteq\mathcal{V}.$$ The notation means that for every partition of experts into S and its complement, no nonzero vector in the desired shared subspace can be invisible to every edge crossing that partition. A sufficient construction is A_{ij}=UC_{ij}U^T, with C_{ij} positive definite on every cut edge. The corresponding regularizer is $$\mathcal{R}_{\mathrm{shared}}=\frac12\sum_{(i,j)\in\mathcal{E}}(z_i-z_j)^TA_{ij}(z_i-z_j),$$ which equals $$\frac12\sum_{(i,j)\in\mathcal{E}}(U^Tz_i-U^Tz_j)^TC_{ij}(U^Tz_i-U^Tz_j).$$ Thus private coordinates in \mathbb{V}^{\perp}=\operatorname{null}(U^T) receive zero coupling when A_{ij} has range exactly \mathbb{V}.

## Key formulas

- $$\mathbb{V}\perp\left(\bigcap_{(i,j)\in\mathcal{E}(S,\bar S)}\operatorname{null}(A_{ij})\right),\qquad\forall S\subseteq\mathcal{V}.$$
- $$A_{ij}=UC_{ij}U^T,\qquad P_{\mathbb{V}}=UU^T,\qquad \mathbb{V}^{\perp}=\operatorname{null}(U^T).$$
- $$\mathcal{R}_{\mathrm{shared}}=\frac12\sum_{(i,j)\in\mathcal{E}}(z_i-z_j)^TA_{ij}(z_i-z_j)=\frac12\sum_{(i,j)\in\mathcal{E}}(U^Tz_i-U^Tz_j)^TC_{ij}(U^Tz_i-U^Tz_j).$$
- $$z_i\leftarrow z_i-\eta_r\nabla_{z_i}\mathcal{L}_i-\eta_s\sum_{j:(i,j)\in\mathcal{E}}A_{ij}(z_i-z_j).$$

## Implementation notes

Integrate the method into a mixture-of-experts transformer or a multi-task graph neural network at the expert output projection, before the router combines expert outputs. Represent each expert's selected feature vector as z_i in R^d and choose an orthonormal shared basis U in R^{d x r}. Parameterize each expert as `W_i = U B_shared + (I-U@U.T) B_i_private` for a low-rank shared matrix B_shared and an expert-specific residual B_i_private, or use the same decomposition on hidden activations. During training, add the exact regularizer R_shared shown in the formulas, with A_{ij}=U C_{ij} U.T; do not penalize private residuals. The update loop is: compute task losses L_i; compute projected differences `q_ij=U.T@(z_i-z_j)`; accumulate `g_i_shared=sum_j C_ij@q_ij`; update each expert with `z_i -= eta_r*grad_L_i + eta_s*U@g_i_shared`. Estimate whether synchronization is structurally possible by sampling cuts S and calculating `lambda_min(sum_(i,j crossing S) C_ij)`; if this is near zero, add an edge or increase its rank rather than merely increasing the regularization coefficient. The first cheap experiment should use a 4-expert, 2-layer transformer on a mixture of four perturbed language-classification tasks, comparing independent experts, standard shared-bottom training, and this shared-private layer at equal parameter count. Sweep r over 10%, 25%, and 50% of channels. Measure average task accuracy, worst-task accuracy, expert diversity in private coordinates, pairwise disagreement in shared coordinates, and routing load balance. Success is improved worst-task accuracy without collapse of private diversity, with shared-coordinate disagreement decreasing while private-coordinate covariance remains substantially larger than in fully shared experts.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a compact shared-private matrix-weighted expert simulation and verified the core mechanism. The contraction sweep matched the predicted factor |1-eta*gamma*lambda| to numerical precision, private-coordinate change stayed below 2e-15, and the stability transition occurred at eta*gamma*lambda_max=2 as predicted. The graph-cut condition behaved correctly, and shared-private coupling reduced shared disagreement while preserving private diversity unlike full sharing.

### Mechanism check

- Verdict: Built a compact shared-private matrix-weighted expert simulation and verified the core mechanism. The contraction sweep matched the predicted factor |1-eta*gamma*lambda| to numerical precision, private-coordinate change stayed below 2e-15, and the stability transition occurred at eta*gamma*lambda_max=2 as predicted. The graph-cut condition behaved correctly, and shared-private coupling reduced shared disagreement while preserving private diversity unlike full sharing.
- Confidence: 9/10
- Limitations: This is a NumPy toy consensus and quadratic optimization experiment, not a transformer, MoE router, or language-classification benchmark. It does not test equal-parameter-count neural models, learned shared bases, noisy or stochastic gradients, routing balance, or scalability beyond four experts.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/253/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/253/bench_report.json)
- [experiment.py](https://synthcore.org/code/253/experiment.py)
- [report.md](https://synthcore.org/code/253/report.md)
- [report_bench_2026-09-02T215545.md](https://synthcore.org/code/253/report_bench_2026-09-02T215545.md)
- [results.json](https://synthcore.org/code/253/results.json)
- [Download all files as ZIP](https://synthcore.org/download/253)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.