# Pareto-Score Dynamic Sparse Masks

- ID: 413
- Canonical URL: https://synthcore.org/idea/413/pareto-score-dynamic-sparse-masks
- API JSON: https://synthcore.org/api/idea/413.json
- API Markdown: https://synthcore.org/api/idea/413.md
- Verification status: unverified
- Source: [arXiv:2607.04765](https://arxiv.org/abs/2607.04765)
- Category: architecture
- Solves: memory, speedup, accuracy
- ML areas: pruning, transformer, mlp
- Math tags: optimization, combinatorics, probability, statistics
- Ratings: usefulness 5/10; difficulty 5/10; novelty 4/10

## Idea description

Replace a neural layer's dense parameters by a real parameter tensor and a binary activity mask, then update the mask using importance scores measured across multiple perturbation intervals. Gates that repeatedly occur in non-dominated accuracy–cost configurations receive larger scores and are more likely to be retained or activated, avoiding uniformly random sparse rewiring.

## Mathematical statement

Equation (2) represents each sparse decision variable as x_c=dec_c    imes mask_c, where dec_c is its real value and mask_c\in\{0,1\} indicates whether it is active. Equation (7) partitions the allowed range of variable c into NVal intervals: Dec_{\cdot c}\in[lower_c+(upper_c-lower_c)(m-1)/NVal,\;lower_c+(upper_c-lower_c)m/NVal] for interval m. The paper evaluates candidate solutions over these intervals and aggregates non-dominated-front information into Score_c, so a variable is rewarded when activating or perturbing it contributes to good Pareto solutions across its whole range. Equation (9) then samples a mask deletion with probability P^C_{j,0}=Score_j/\sum_{idx\in C_1}Score_{idx}, where C_1 is the set of candidate active variables differing between two parent masks. For neural networks, let g_j be a gate for structured unit j, w_j its real weights, and x_j=w_jg_j. Define objectives F=(validation loss, active-cost), with active-cost equal to FLOPs or parameter bytes. Estimate Score_j by evaluating several perturbation magnitudes in Eq. (7), ranking candidates by non-dominated fronts, and normalizing the resulting front counts to positive scores. The exact score aggregation is not present in the extracted text, so the implementation should use the stated front-count aggregation principle and report the selected normalization.

## Key formulas

- $$\mathbf{x}=\big(dec_{1}\times mask_{1},\;\dots,\;dec_{D}\times mask_{D}\big),\qquad mask_c\in\{0,1\}.$$
- $$Dec_{\cdot c}\in\left[lower_{c}+\frac{(upper_{c}-lower_{c})(m-1)}{NVal},\;lower_{c}+\frac{(upper_{c}-lower_{c})m}{NVal}\right].$$
- $$P_{j,0}^{C}=\frac{Score_{j}}{\sum_{idx\in C_{1}}Score_{idx}}.$$
- $$F(g)=\left(\mathcal{L}_{val}(g),\;\mathrm{Cost}(g)\right),\qquad x_j=w_jg_j.$$

## Implementation notes

Integrate this into a dynamic-sparse MLP or Transformer feed-forward layer at the structured-unit level: each neuron, attention head, or FFN block has real weights w_j and a binary gate g_j, and the effective output uses w_jg_j. Keep a target active fraction k/D and periodically rewire a small fraction r of gates rather than changing masks every optimizer step. For each candidate unit j, create NVal=4 perturbation trials by temporarily scaling its contribution by s_m=lower+(upper-lower)m/NVal, or by evaluating the unit at four interpolation strengths between zero and its current activation. For every trial, measure validation loss and incremental FLOPs; collect all candidates and assign Pareto-front rank, where rank 0 is non-dominated. Define a positive score such as Score_j=1/(1+mean_m FrontRank_{j,m}), with epsilon added before normalization. To delete active units, sample from the active disagreement set using Eq. (9), but invert the score if high score means importance: deletion probability is proportional to 1/(Score_j+epsilon). To activate inactive units, sample proportional to Score_j. After rewiring, reset newly active weights to their stored values or to zero followed by a short warm-up, and train with the straight-through estimator for g_j in the forward pass and gradient mask g_j in the backward pass. Compute front ranks and costs empirically; the interval partition and sampling probabilities come directly from the paper. First test a 2-layer MLP on CIFAR-10 or a 6-layer Transformer on WikiText-2 at 50% structured sparsity, comparing random RigL-style rewiring and magnitude pruning. Success is higher accuracy at equal active FLOPs, or faster validation-loss descent at the same wall-clock budget.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.