Tree-motif anti-collapse masks
Implementation & benchmark of arXiv:2607.05384 — Well-invertible column subsets of sparse matrices are rare
Source paper: Well-invertible column subsets of sparse matrices are rare arXiv:2607.05384 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Use the paper's explicit tree support pattern as a cheap certificate that a sparse neural linear map contains a nearly singular submatrix. During mask construction or rewiring, penalize root-row-child configurations with many disjoint child branches, or increase overlap and row degree locally when such a configuration is detected. The goal is to prevent sparse MLP, projection, or MoE expert matrices from developing directions that are almost annihilated by the layer.
Formulas
Mathematical statement
For a matrix M with rows as left vertices and columns as right vertices in its bipartite support graph, consider a root column j_star connected to d rows i_1,...,i_d. Suppose each row i_l has m child columns c_{l,1},...,c_{l,m}, and the child branches have disjoint outer-left supports except for their connection to i_l; in the unit-support pattern used in the paper, the outer layer has d m (d-1) nonzero coordinates. Define v on columns by v_{j_star}=1 and choose |v_{c_{l,a}}|=1/m with signs so the root-column contribution cancels on rows i_l. Then the only remaining nonzero coordinates of M_Omega v are on the outer left layer, each with magnitude 1/m, giving ||M_Omega v||_2 <= sqrt(d m (d-1))/m <= d/sqrt(m). Since ||v||_2 >= 1, the smallest singular value satisfies s_min(M_Omega) <= d/sqrt(m). Here M_Omega is the submatrix induced by the detected support pattern, d is the number of root-adjacent rows, m is the number of child columns per row, and s_min(A)=min_{||z||_2=1}||Az||_2. The graph interpretation also gives |supp M_.j intersect supp M_.j'| >= 2 iff the two columns lie on a 4-cycle with two common row neighbors; suppressing excessive 4-cycle incidence or tree motifs removes the structural source of these witnesses.
Implementation notes
Integrate this into the mask-generation and mask-rewiring code for a sparse linear layer W in an MLP, attention projection, or MoE expert; operate on the binary support mask G=1[W != 0], not on activations. First normalize each row to a fixed nonzero count if the baseline uses row-balanced sparsity. Detect candidate motifs by sampling root columns j_star, collecting their neighboring rows i_l, then collecting m child columns from each row. Retain candidates whose child-column neighborhoods outside the root rows are approximately disjoint; exact disjointness is checked with set intersections. For every candidate, construct the paper witness v by setting v[j_star]=1 and v[c_{l,a}]=-1/m for unit-magnitude weights, or use v[c_{l,a}]=-W[i_l,j_star]/(m W[i_l,c_{l,a}]) followed by normalization for weighted matrices. Estimate the certificate q=d/sqrt(m), and add either a mask penalty equal to the sum of q^2 over candidates or a rewiring rule that removes one edge from the largest-q motif and adds it to a low-degree row or column pair. Also count 4-cycles using common-neighbor intersections and optionally constrain their maximum incidence. The exact paper computation is the witness bound q; empirical quantities are motif counts, sampled minimum singular values, gradient norms, and Jacobian condition estimates. Start with a 2-layer sparse MLP on CIFAR-10 or a 6-layer Transformer on a small language corpus, comparing random fixed masks, magnitude pruning, and the anti-collapse mask at identical parameter count and FLOPs. Success means fewer sampled submatrices with s_min below 1e-3, lower gradient anisotropy, slower loss spikes, and equal-or-better validation accuracy at the same sparsity.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.