Boundary-Depth Sparsity

Integrate the regularizer after a ReLU, Softplus, or absolute-value activation in a CNN, before the next convolution. Let a[b,c,h,w]≥0 be the feature tensor and let Ω be the H×W grid for one channel. First compute the soft zero membership q=sigmoid((τ−a)/ε), where τ is a small activation threshold, ε controls softness, and δ=10^-6 prevents division by zero. Approximate the zero-phase boundary by b=q(1−q)+δ; this is large for activations near the threshold and small in confidently active or confidently zero pixels. Compute the soft distance d at every pixel with the log-sum-exp soft minimum shown above, using a separable or truncated Gaussian neighborhood for efficiency. Choose F(d)=d^p with p=1 or 2, or F(d)=log(1+d), and add the negative depth reward to the task loss with coefficient λ. Pseudocode is: a=relu(conv(x)); q=sigmoid((τ−a)/ε); b=q*(1−q)+δ; d=soft_distance_to_boundary(b,rho); L=L_task−λ*mean(q*F(d)); backpropagate L and update weights. Anneal ε from 0.2 to 0.05 and λ from zero during the first few epochs to avoid immediate all-zero collapse. The paper contributes the depth-weighted zero-phase objective; the soft mask, boundary proxy, and distance transform are engineering approximations. Estimate sparsity, connected-component size, mean zero-pixel depth, validation accuracy, and hardware runtime. The first experiment should train ResNet-18 on CIFAR-10 with this term versus identical training with L1 activation regularization, group lasso, and no regularizer. At equal accuracy, success means fewer fragmented nonzeros, larger zero blocks, and measurable speedup after compiling block-sparse or spatial-skipping convolution; a useful target is 20–30% fewer active spatial blocks with less than 0.5 percentage-point accuracy loss.