Recursive variation-norm regularization

Integrate the method into the hidden MLP layers, specifically the incoming affine tensor of every hidden unit and the final readout, during training. Use a dense network with width 128 and depth 2, 4, or 8. Parameterize each unit as \(h_{l,j}=\sigma_{s_{l,j}}(g_{l,j})\), where \(g_{l,j}=u_{l,j}^{\top}h_{l-1}+b_{l,j}\), and parameterize \(s_{l,j}=\operatorname{softplus}(r_{l,j})+10^{-3}\). In every forward pass, compute a recursive cost: initialize \(q_{0,j}=0\); for each layer compute \(q_{l,j}=q_{l-1,j}+\|u_{l,j}\|_1+|b_{l,j}|\); for output coefficients \(a_j\), set \(V=\sum_j |a_j|(1+q_{L,j})+|b_{out}|\). Optimize cross-entropy or MSE plus \(\lambda V\). Replace absolute values by \(\sqrt{z^2+\epsilon}\), with \(\epsilon=10^{-8}\), if direct subgradients are unstable. The exact paper quantity is the normalized activation in the first formula; the recursive finite-width \(V\) is an explicit engineering surrogate for its recursively defined ℓ1-bounded function space. Do not estimate the representer-theorem constant \(C\) during training. Sweep \(\lambda\) over \(10^{-5},10^{-4},10^{-3},10^{-2}\), and compare against the same models with AdamW/L2 decay, matching validation loss and optimizer settings. First run MNIST or Fashion-MNIST with ReLU, GELU, and SiLU. Record training and validation accuracy, learned \(V\), parameter norms, gradient variance, activation magnitudes, and loss at equal optimizer steps. Test the structural prediction separately on one-dimensional regression: train on samples of \(y=\sin(2\pi kx)\) and evaluate Fourier amplitude on held-out frequencies, measuring the largest frequency learned at fixed \(V\). A positive result is lower validation error, lower gradient variance, or equal accuracy with smaller recursive variation cost than L2. The depth-saturation prediction is supported if deeper ReLU models do not increase the maximum learned frequency at matched \(V\), while non-homogeneous GELU or SiLU models do. A negative result is underfitting or no improvement over L2 at matched validation loss.