Water-Filled Block-Sparse Neural Connectivity

1. Integration point: apply the construction to a transformer attention layer, an MoE expert communication layer, or a graph-neural-network message-passing layer. Partition modules into two classes using an observable noise score, such as routing entropy, gradient variance, activation corruption sensitivity, or disagreement under data augmentation. Replace dense module-to-module communication by a two-block soft mask with probabilities w_11, w_12, and w_22. At inference, sample a fixed sparse mask or retain the highest-scoring edges subject to the same density. 2. Pseudocode: initialize the three block probabilities at target density rho; run minibatches with differentiable or straight-through Bernoulli masks; estimate F and each marginal g_ij by backpropagation or finite differences; update w_ij with clip(w_ij+gamma*(g_ij-lambda)); find lambda by bisection so the weighted density constraint is exact; periodically resample masks and recompute class scores. 3. Taken from the paper are the graphon relaxation, fixed-density constraint, SBM reduction, logit response analogy, and water-filling/KKT rule. Estimated empirically are alpha_i, beta_i, utilities, and gradients. 4. First experiment: train a small transformer on CIFAR-10 patches or WikiText-2 and compare dense attention, random block sparsity, unconstrained learned sparsity, and water-filled sparsity at equal FLOPs. Sweep rho and record validation loss, gradient variance, and communication disagreement. Prediction: at least one of w_11, w_12, or w_22 reaches near-zero or near-one, and performance versus rho has a kink when a block enters or leaves saturation. The selected active block ranking should agree with measured marginal utilities in at least 80% of runs; absence of saturation or kinks falsifies the mechanism.