Rank-safe Bernoulli layer initialization
Implementation & benchmark of arXiv:2607.00495 — Rank deficiency of Bernoulli random matrices for growing corank
Source paper: Rank deficiency of Bernoulli random matrices for growing corank arXiv:2607.00495 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Use the Bernoulli corank asymptotic to choose sparsity for binary or sparse linear layers and reject initial matrices with excessive numerical rank deficiency. The layer should also explicitly prevent zero columns, because the paper's probability law indicates that zero-column events are a leading mechanism behind large corank in the sparse regime.
Formulas
Mathematical statement
For an n x n matrix A with independent entries A_{ij}~Bernoulli(p), p<=1/2, and integer k satisfying k=o(\sqrt{\log n}), the paper proves P(corank(A)>=k)=(1-p+o_n(1))^{kn}, where corank(A)=n-rank(A), rank is the dimension of the column space, and o_n(1) tends to zero as n tends to infinity. The paper also uses the exact equivalence P(det(A)=0)=P(\exists x\in\mathbb{R}^n\setminus\{0\}:\|Ax\|_2=0), with \|\cdot\|_2 the Euclidean norm. The transferable mechanism is that a zero column has probability (1-p)^n, and k independent zero-column-type defects produce the leading scale (1-p)^{kn}. For a rectangular neural weight matrix W\in\{0,1\}^{m imes d}, the square theorem is not a formal guarantee; use the analogous proxy (1-p)^m per zero input column and measure actual rank numerically. A target upper bound \delta for k structural defects gives the design rule (1-p)^{mk}\leq\delta, equivalently p\geq 1-\delta^{1/(mk)}.
Implementation notes
Integrate this at initialization of binary or sparse fully connected layers, convolutional pointwise matrices, and low-bit MLP blocks. Let W have shape m x d, with entries sampled independently as Bernoulli(p) and optionally mapped to signed weights W_{ij}=s(2B_{ij}-1), where B_{ij}~Bernoulli(p); retain a separate scale s=1/\sqrt{mp} so activation variance is comparable across p. First choose a desired structural-defect tolerance delta and an estimated defect count k_target, then set p=max(p_min, 1-delta**(1/(m*k_target))). Do not interpret this rectangular rule as a theorem: it is the paper's square-matrix asymptotic used as a conservative proxy. Sample W, count zero columns, and estimate numerical rank by computing singular values for small layers or by randomized range finding for large layers. If any column is zero or the estimated corank exceeds k_target, resample only the affected columns; alternatively force each column to contain at least one nonzero entry and then rescale. Pseudocode: sample B; while zero_columns(B) or numerical_corank(W,tau)>k_target: resample offending columns; scale W; run a forward variance check. For training, compare this initializer against standard iid Bernoulli, dense Gaussian, and orthogonal initialization in a 2-layer MLP on MNIST or Fashion-MNIST, plus a width- and sparsity-matched binary network. Measure initial numerical rank, fraction of dead hidden units, gradient norm dispersion, validation accuracy, and loss after a fixed number of updates. The expected signal is fewer dead channels and more stable early optimization at the same parameter count; if rank screening adds overhead without improving these metrics, reject the method.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.