Shared-Private Matrix-Weighted Expert Layers

Integrate the method into a mixture-of-experts transformer or a multi-task graph neural network at the expert output projection, before the router combines expert outputs. Represent each expert's selected feature vector as z_i in R^d and choose an orthonormal shared basis U in R^{d x r}. Parameterize each expert as W_i = U B_shared + (I-U@U.T) B_i_private for a low-rank shared matrix B_shared and an expert-specific residual B_i_private, or use the same decomposition on hidden activations. During training, add the exact regularizer R_shared shown in the formulas, with A_{ij}=U C_{ij} U.T; do not penalize private residuals. The update loop is: compute task losses L_i; compute projected differences q_ij=U.T@(z_i-z_j); accumulate g_i_shared=sum_j C_ij@q_ij; update each expert with z_i -= eta_r*grad_L_i + eta_s*U@g_i_shared. Estimate whether synchronization is structurally possible by sampling cuts S and calculating lambda_min(sum_(i,j crossing S) C_ij); if this is near zero, add an edge or increase its rank rather than merely increasing the regularization coefficient. The first cheap experiment should use a 4-expert, 2-layer transformer on a mixture of four perturbed language-classification tasks, comparing independent experts, standard shared-bottom training, and this shared-private layer at equal parameter count. Sweep r over 10%, 25%, and 50% of channels. Measure average task accuracy, worst-task accuracy, expert diversity in private coordinates, pairwise disagreement in shared coordinates, and routing load balance. Success is improved worst-task accuracy without collapse of private diversity, with shared-coordinate disagreement decreasing while private-coordinate covariance remains substantially larger than in fully shared experts.