Criticality-controlled sparsity schedule
Implementation & benchmark of arXiv:2608.12295 — Finite-depth scaling and an exact Bernoulli-leaf identity for the min-plus process on the binary tree
Source paper: Finite-depth scaling and an exact Bernoulli-leaf identity for the min-plus process on the binary tree arXiv:2608.12295 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Use the min-plus phase transition as a training-time controller: begin near p = 1/2 to preserve the initial active-state fraction across depth, then move above or below criticality to deliberately remove or create sparse pathways. The controller uses a measurable state variable, the activation zero fraction, rather than an arbitrary regularization coefficient.
Formulas
Mathematical statement
Let z_n be the empirical fraction of zero activations after level n in a stochastic hierarchical network. The mean-field state update is f_p(z) = 2(1-p)z + (2p-1)z^2, and the local multiplier is f'_p(z) = 2(1-p) + 2(2p-1)z. At p = 1/2, f_p(z) = z, so sparsity is depth-invariant. Away from criticality, z_(n+1) - z_n = (1-2p)z_n(1-z_n). A feedback controller can therefore adjust p using the measured zero fraction, keeping the network near criticality or intentionally driving it toward a sparse absorbing regime.
Implementation notes
Integration point: apply the controller to the merge probability of a stochastic hierarchical block, or to a layerwise schedule shared by all merges at training iteration t. Use ReLU, Softplus followed by thresholding, or another nonnegative activation so that exact zeros are meaningful. Pseudocode: at iteration t, estimate zhat_t as the fraction of zero coordinates over the current minibatch and all tree levels; update p_(t+1) = clip(p_t + kappa(zhat_t - z_star), p_min, p_max); sample merge gates with the updated p; backpropagate the task loss and optionally add a weak penalty lambda times (zhat_t - z_star)^2. Initialize with p = 0.5. The paper supplies the mean-field recursion, fixed points, and critical boundary. The implementation estimates finite-width correlations, actual sparsity, gradient survival, and computational savings. First experiment: use a six-level tree MLP on MNIST with target sparsities z_star equal to 0.1, 0.3, and 0.5, comparing fixed p = 0.5, fixed p = 0.7, and feedback control. Log z_n, gradient norm by depth, wall-clock cost, and accuracy. The predicted signature is a sign change in z_(n+1) - z_n at p = 0.5, with near-zero drift at criticality. The measured transition should agree with 0.5 plus or minus 0.05, and a sparse regime should reduce active merge cost without exponential gradient decay.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.