# Variation-budget neuron growth

- ID: 2987
- Canonical URL: https://synthcore.org/idea/2987/variation-budget-neuron-growth
- API JSON: https://synthcore.org/api/idea/2987.json
- API Markdown: https://synthcore.org/api/idea/2987.md
- Verification status: unverified
- Source: [arXiv:2609.00680](https://arxiv.org/abs/2609.00680)
- Category: architecture
- Solves: memory, scalability, generalization
- ML areas: mlp, pruning, memory, initialization
- Math tags: measure-theory, functional-analysis, approximation-theory, harmonic-analysis
- Ratings: usefulness 5/10; difficulty 5/10; novelty 5/10

## Idea description

Treat a finite shallow network as a discrete signed measure over ridge atoms and grow or prune neurons according to their contribution to total variation. This turns width selection into an atomic approximation procedure: add neurons correlated with the current residual and remove coefficients that consume budget without contributing materially.

## Mathematical statement

For \(\omega_i\in\mathbb{S}^{d-1}\), offsets \(b_i\in[-c,c]\), and coefficients \(a_i\in\mathbb{R}\), a finite network corresponds to \(\mu=\sum_{i=1}^ma_i\delta_{(\omega_i,b_i)}\), giving \(f_\mu(x)=\sum_{i=1}^ma_i(\omega_i\cdot x-b_i)_+^k\). The measure's total variation is \(\|\mu\|_{\mathrm{TV}}=\sum_i|a_i|\), while the paper's atomic variation norm is the infimum of this quantity over all measures representing the same function: \(\|f\|_{\mathcal{L}_1(\mathcal{D})}=\inf_{\mu:f=f_\mu}\|\mu\|_{\mathrm{TV}}\). The reverse embedding for \(0<p<1\) is \(\mathcal{L}_1(\mathcal{D})\hookrightarrow B^{k+1}_{p,2}(\Omega)\), so bounded variation imposes Besov smoothness \(k+1\) and fine index 2. We use \(V=\sum_i|a_i|\) as an explicit complexity and memory-control variable. New atoms are selected by correlation with residuals over the same sphere-and-offset dictionary.

## Key formulas

- $$f_\mu(x)=\int_{\mathbb{S}^{d-1}\times[-c,c]}\sigma_k(\omega\cdot x-b)\,d\mu(\omega,b).$$
- $$\mu=\sum_{i=1}^{m}a_i\delta_{(\omega_i,b_i)},\qquad f_\mu(x)=\sum_{i=1}^{m}a_i(\omega_i\cdot x-b_i)_+^k,\qquad \|\mu\|_{\mathrm{TV}}=\sum_{i=1}^{m}|a_i|.$$
- $$\|f\|_{\mathcal{L}_1(\mathcal{D})}:=\inf_{\mu:f=f_\mu}\|\mu\|_{\mathrm{TV}},\qquad \mathcal{L}_1(\mathcal{D})\hookrightarrow B^{k+1}_{p,2}(\Omega)\quad(0<p<1).$$
- $$\max_{(\omega,b)\in\mathbb{S}^{d-1}\times[-c,c]}\left|\frac{1}{n}\sum_{r=1}^{n}e_r\,\sigma_k(\omega\cdot x_r-b)\right|.$$

## Implementation notes

Replace a dense shallow layer by an active set of neurons \((\omega_i,b_i,a_i)\). Maintain minibatch residuals \(e_r=f_\theta(x_r)-y_r\). Every \(T\) optimizer steps, sample \(K\) random unit directions and offsets, evaluate the absolute residual correlation \(c(\omega,b)=|n^{-1}\sum_re_r\sigma_k(\omega\cdot x_r-b)|\), and add the best atom. Optionally refine its \((\omega,b)\) by a few gradient-ascent steps on \(c\). Initialize its coefficient as \(a=\eta_a\operatorname{sign}(n^{-1}\sum_re_r\sigma_k(\omega\cdot x_r-b))\), then jointly optimize all active parameters. Every \(T_{prune}\) steps, delete atoms with \(|a_i|<\tau\), and enforce a total variation budget \(V=\sum_i|a_i|\le V_{max}\) by soft-thresholding or projection onto the \(\ell^1\) ball. The paper supplies the dictionary, TV complexity, and reverse Besov regularity claim; sampling size, coefficient step, pruning threshold, and budget are empirical. First compare active-set growth against a fixed width-256 shallow MLP and magnitude pruning on 2D regression with smooth and localized targets. Log validation error versus active neuron count, \(V\), wall-clock time, and inference memory. Success is reaching a target error with fewer active neurons and lower memory, while the smooth target should exhibit smaller final variation and the localized target should require additional atoms rather than being incorrectly over-pruned.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.