# Criticality-controlled sparsity schedule

- ID: 2177
- Canonical URL: https://synthcore.org/idea/2177/criticality-controlled-sparsity-schedule
- API JSON: https://synthcore.org/api/idea/2177.json
- API Markdown: https://synthcore.org/api/idea/2177.md
- Verification status: unverified
- Source: [arXiv:2608.12295](https://arxiv.org/abs/2608.12295)
- Category: training
- Solves: stability, regularization, speedup
- ML areas: optimizer, scheduler, training-dynamics, regularization, mlp
- Math tags: statistical-mechanics, dynamical-systems, probability, bifurcations
- Ratings: usefulness 6/10; difficulty 5/10; novelty 9/10

## Idea description

Use the min-plus phase transition as a training-time controller: begin near p = 1/2 to preserve the initial active-state fraction across depth, then move above or below criticality to deliberately remove or create sparse pathways. The controller uses a measurable state variable, the activation zero fraction, rather than an arbitrary regularization coefficient.

## Mathematical statement

Let z_n be the empirical fraction of zero activations after level n in a stochastic hierarchical network. The mean-field state update is f_p(z) = 2(1-p)z + (2p-1)z^2, and the local multiplier is f'_p(z) = 2(1-p) + 2(2p-1)z. At p = 1/2, f_p(z) = z, so sparsity is depth-invariant. Away from criticality, z_(n+1) - z_n = (1-2p)z_n(1-z_n). A feedback controller can therefore adjust p using the measured zero fraction, keeping the network near criticality or intentionally driving it toward a sparse absorbing regime.

## Key formulas

- $f_p(z) = 2(1-p)z + (2p-1)z^2$
- $f'_p(z) = 2(1-p) + 2(2p-1)z$
- $z_{n+1}-z_n = (1-2p)z_n(1-z_n)$
- $p_{t+1} = clip(p_t + kappa(zhat_t-z_star), p_min, p_max)$

## Implementation notes

Integration point: apply the controller to the merge probability of a stochastic hierarchical block, or to a layerwise schedule shared by all merges at training iteration t. Use ReLU, Softplus followed by thresholding, or another nonnegative activation so that exact zeros are meaningful. Pseudocode: at iteration t, estimate zhat_t as the fraction of zero coordinates over the current minibatch and all tree levels; update p_(t+1) = clip(p_t + kappa(zhat_t - z_star), p_min, p_max); sample merge gates with the updated p; backpropagate the task loss and optionally add a weak penalty lambda times (zhat_t - z_star)^2. Initialize with p = 0.5. The paper supplies the mean-field recursion, fixed points, and critical boundary. The implementation estimates finite-width correlations, actual sparsity, gradient survival, and computational savings. First experiment: use a six-level tree MLP on MNIST with target sparsities z_star equal to 0.1, 0.3, and 0.5, comparing fixed p = 0.5, fixed p = 0.7, and feedback control. Log z_n, gradient norm by depth, wall-clock cost, and accuracy. The predicted signature is a sign change in z_(n+1) - z_n at p = 0.5, with near-zero drift at criticality. The measured transition should agree with 0.5 plus or minus 0.05, and a sparse regime should reduce active merge cost without exponential gradient decay.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.