Unverified 2026

Criticality-controlled sparsity schedule

Implementation & benchmark of arXiv:2608.12295 — Finite-depth scaling and an exact Bernoulli-leaf identity for the min-plus process on the binary tree

Usefulness6/10
Difficulty5/10
Novelty9/10

Source paper: Finite-depth scaling and an exact Bernoulli-leaf identity for the min-plus process on the binary tree arXiv:2608.12295 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use the min-plus phase transition as a training-time controller: begin near p = 1/2 to preserve the initial active-state fraction across depth, then move above or below criticality to deliberately remove or create sparse pathways. The controller uses a measurable state variable, the activation zero fraction, rather than an arbitrary regularization coefficient.

Formulas

$f_p(z) = 2(1-p)z + (2p-1)z^2$
$f'_p(z) = 2(1-p) + 2(2p-1)z$
$z_{n+1}-z_n = (1-2p)z_n(1-z_n)$
$p_{t+1} = clip(p_t + kappa(zhat_t-z_star), p_min, p_max)$

Mathematical statement

Let z_n be the empirical fraction of zero activations after level n in a stochastic hierarchical network. The mean-field state update is f_p(z) = 2(1-p)z + (2p-1)z^2, and the local multiplier is f'_p(z) = 2(1-p) + 2(2p-1)z. At p = 1/2, f_p(z) = z, so sparsity is depth-invariant. Away from criticality, z_(n+1) - z_n = (1-2p)z_n(1-z_n). A feedback controller can therefore adjust p using the measured zero fraction, keeping the network near criticality or intentionally driving it toward a sparse absorbing regime.

Implementation notes

Integration point: apply the controller to the merge probability of a stochastic hierarchical block, or to a layerwise schedule shared by all merges at training iteration t. Use ReLU, Softplus followed by thresholding, or another nonnegative activation so that exact zeros are meaningful. Pseudocode: at iteration t, estimate zhat_t as the fraction of zero coordinates over the current minibatch and all tree levels; update p_(t+1) = clip(p_t + kappa(zhat_t - z_star), p_min, p_max); sample merge gates with the updated p; backpropagate the task loss and optionally add a weak penalty lambda times (zhat_t - z_star)^2. Initialize with p = 0.5. The paper supplies the mean-field recursion, fixed points, and critical boundary. The implementation estimates finite-width correlations, actual sparsity, gradient survival, and computational savings. First experiment: use a six-level tree MLP on MNIST with target sparsities z_star equal to 0.1, 0.3, and 0.5, comparing fixed p = 0.5, fixed p = 0.7, and feedback control. Log z_n, gradient norm by depth, wall-clock cost, and accuracy. The predicted signature is a sign change in z_(n+1) - z_n at p = 0.5, with near-zero drift at criticality. The measured transition should agree with 0.5 plus or minus 0.05, and a sparse regime should reduce active merge cost without exponential gradient decay.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.