Pareto-Score Dynamic Sparse Masks

Integrate this into a dynamic-sparse MLP or Transformer feed-forward layer at the structured-unit level: each neuron, attention head, or FFN block has real weights w_j and a binary gate g_j, and the effective output uses w_jg_j. Keep a target active fraction k/D and periodically rewire a small fraction r of gates rather than changing masks every optimizer step. For each candidate unit j, create NVal=4 perturbation trials by temporarily scaling its contribution by s_m=lower+(upper-lower)m/NVal, or by evaluating the unit at four interpolation strengths between zero and its current activation. For every trial, measure validation loss and incremental FLOPs; collect all candidates and assign Pareto-front rank, where rank 0 is non-dominated. Define a positive score such as Score_j=1/(1+mean_m FrontRank_{j,m}), with epsilon added before normalization. To delete active units, sample from the active disagreement set using Eq. (9), but invert the score if high score means importance: deletion probability is proportional to 1/(Score_j+epsilon). To activate inactive units, sample proportional to Score_j. After rewiring, reset newly active weights to their stored values or to zero followed by a short warm-up, and train with the straight-through estimator for g_j in the forward pass and gradient mask g_j in the backward pass. Compute front ranks and costs empirically; the interval partition and sampling probabilities come directly from the paper. First test a 2-layer MLP on CIFAR-10 or a 6-layer Transformer on WikiText-2 at 50% structured sparsity, comparing random RigL-style rewiring and magnitude pruning. Success is higher accuracy at equal active FLOPs, or faster validation-loss descent at the same wall-clock budget.