# Level-Set Balanced Sparse Mixer

- ID: 2921
- Canonical URL: https://synthcore.org/idea/2921/level-set-balanced-sparse-mixer
- API JSON: https://synthcore.org/api/idea/2921.json
- API Markdown: https://synthcore.org/api/idea/2921.md
- Verification status: unverified
- Source: [arXiv:2607.23017](https://arxiv.org/abs/2607.23017)
- Category: architecture
- Solves: stability, accuracy, memory
- ML areas: mlp, transformer, regularization, inference-speedup
- Math tags: random-matrix, probability, linear-algebra, combinatorics
- Ratings: usefulness 5/10; difficulty 6/10; novelty 7/10

## Idea description

Partition activations into dyadic magnitude bands and allocate sparse connectivity separately to heavy and diffuse coordinates. Protect high-magnitude coordinates with more reliable connections while using randomized flat connectivity for the many small coordinates, keeping the total number of nonzeros fixed.

## Mathematical statement

For x in R^n, the paper defines the dyadic level set I_j(x) := {i in [n] : 2^{-j} < |x_i| <= 2^{-j+1}}. It splits the sparse action into a Tall part and a Flat part and uses the deterministic inequality ||Pi x||_2 <= ||T_L^can(Pi,x)x||_2 + ||F_L^can(Pi,x)x||_2. For a 1/2-net N_V contained in V intersected with the unit sphere, it further gives ||Pi U_V|| <= 2 sup_{z in N_V} ||T_L^can(Pi,z)z||_2 + 2 sup_{z in N_V} ||F_L^can(Pi,z)z||_2. The surrounding discussion identifies heavy columns and coordinates with the Tall contribution, handled by row concentration, and light coordinates and couples with the Flat contribution, controlled through entropy of level-set configurations. The neural adaptation freezes a level-set-aware mask after calibration, avoiding a fully input-dependent mask during training. The exact theorem does not apply to adaptive neural masks, so this is a falsifiable architectural heuristic rather than a direct guarantee.

## Key formulas

- $$I_j(x):=\{i\in[n]:2^{-j}<\left|x_i\right|\leq 2^{-j+1}\}.$$
- $$\left\lVert\Pi x\right\rVert_2\leq\left\lVert\mathcal{T}_L^{\rm can}(\Pi,x)x\right\rVert_2+\left\lVert\mathcal{F}_L^{\rm can}(\Pi,x)x\right\rVert_2.$$
- $$\left\lVert\Pi U_V\right\rVert\leq 2\sup_{z\in\mathcal{N}_V}\left\lVert\mathcal{T}_L^{\rm can}(\Pi,z)z\right\rVert_2+2\sup_{z\in\mathcal{N}_V}\left\lVert\mathcal{F}_L^{\rm can}(\Pi,z)z\right\rVert_2.$$
- $$\widetilde W_{ij}=\frac{b_{ij}\xi_{ij}}{\sqrt{k p_{j_i}}},\qquad p_j=\min\left\{1,p_0 2^{2j}\right\}.$$

## Implementation notes

Apply the mechanism to a token-mixing or MLP projection y = W h, with h in R^n and W in R^{k x n}. During calibration, compute a_i = |h_i| + 10^{-8} and assign each coordinate to a dyadic band j_i = floor(-log_2(a_i)); clip j_i to [-8,8]. Define heavy coordinates as the largest five percent by magnitude, or as coordinates above a fixed percentile threshold, and define all others as flat. Give every output row a fixed heavy-coordinate budget d_H and a flat-coordinate budget d_F, with d_H + d_F equal to the nonzero budget of the unstructured sparse baseline. Freeze the resulting mask after calibration for deployment. Use W_ij = b_ij xi_ij/sqrt(k p_{j_i}), where p_j = min(1, p_0 2^{2j}) after choosing the sign convention so larger j corresponds to smaller magnitudes; alternatively estimate p_j from calibration by making p_j proportional to the inverse conditional second moment E[a_i^2 | j_i = j], clipped to [p_F,1]. Pseudocode: compute magnitude bands; select heavy coordinates; sample or assign protected heavy connections; sample flat connections; apply per-band scaling; compute y. The paper contributes the level-set and Tall-plus-Flat decomposition; density schedules and thresholds are engineering choices. First test a width-512 Transformer on WikiText-2 or a small ViT on CIFAR-100 against an unstructured mask with identical parameters, nonzeros, optimizer, and measured FLOPs. Pre-register a 20 percent reduction in the 99.9th-percentile amplification ratio ||Wh||/||h||, lower variance of the heavy contribution ||W h_heavy||/||Wh||, and a smaller loss increase when flat coordinates are ablated instead of heavy coordinates. The mechanism ablation is a same-density global random mask with global 1/sqrt(kp) scaling and no level-set allocation. Falsify the idea if frozen level-set masks do not reduce amplification tails by at least 10 percent or if the result vanishes against a magnitude-aware but non-dyadic baseline.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.