# Robustness-capacity feasibility controller

- ID: 558
- Canonical URL: https://synthcore.org/idea/558/robustness-capacity-feasibility-controller
- API JSON: https://synthcore.org/api/idea/558.json
- API Markdown: https://synthcore.org/api/idea/558.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.07778](https://arxiv.org/abs/2607.07778)
- Category: training
- Solves: stability, generalization, accuracy
- ML areas: mlp, regularization, training-dynamics
- Math tags: geometry, probability, approximation-theory
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Use the paper's lower bound as a feasibility test for robust interpolation: if a model is asked to fit below the estimated noise floor while maintaining a small Lipschitz constant, automatically increase effective width or relax the fit target. This prevents optimization from wasting compute on an impossible low-sensitivity solution and provides a principled width schedule for noisy regression or classification.

## Mathematical statement

The network class is $\mathcal{N}_{m}=\{f(x)=\sum_{k=1}^{m}a_k\psi(\langle w_k,x\rangle+b_k)+\langle v,x\rangle+c\}$, where $m$ is hidden width, $a_k,b_k,c\in\mathbb{R}$, and $w_k,v\in\mathbb{R}^d$. For labels with conditional noise variance $\sigma^2>0$, the theorem states that, with high probability for spherical data in $d\geq 3$ or isotropic Gaussian data, any width-$m$ piecewise-linear network with empirical fitting error $\varepsilon$ below the noise floor obeys $\operatorname{Lip}(f)\geq c\varepsilon\sqrt{n/[\bar m\log(C\bar mnd/\varepsilon)]}$, where $n$ is sample count, $\bar m=(K-1)m+1$ depends on the activation's number $K$ of linear pieces, and $c,C>0$ are constants. For ReLU, $K=2$ and hence $\bar m=m+1$. The controller uses this necessary bound as $L_{\min}(m,\varepsilon)$; it does not claim the bound is sufficient for learnability.

## Key formulas

- $$\mathcal{N}_{m}=\left\{f(x)=\sum_{k=1}^{m}a_k\,\psi(\langle w_k,x\rangle+b_k)+\langle v,x\rangle+c\right\}.$$
- $$\operatorname{Lip}(f)\geq c\,\varepsilon\sqrt{\frac{n}{\bar m\log(C\bar mnd/\varepsilon)}},\qquad \bar m=(K-1)m+1.$$
- $$L_{\min}(m,\varepsilon)=c_0\,\varepsilon\sqrt{\frac{n}{(m+1)\log(C_0(m+1)nd/\max(\varepsilon,\delta))}}\quad\text{for ReLU}.$$

## Implementation notes

Integrate this into training of a two-layer ReLU MLP immediately after each validation interval. Use a minibatch or held-out probe set to estimate the current fitting error $\hat\varepsilon$ and the input Lipschitz constant. Estimate $\operatorname{Lip}(f)$ by power iteration on the input Jacobian, $L_{\mathrm{est}}=\max_{x\in B}\|J_f(x)\|_2$, using 32--128 random probe points plus training examples; for scalar output, the Jacobian is a row vector. Estimate the noise floor from repeated labels, a known corruption rate, or a robust residual variance estimator. Compute $L_{\min}=c_0\hat\varepsilon\sqrt{n/[(m+1)\log(C_0(m+1)nd/\max(\hat\varepsilon,\delta))]}$, with conservative tunable constants $c_0,C_0$ and numerical floor $\delta$. Pseudocode: (1) train normally for an interval; (2) measure $\hat\varepsilon$ and $L_{\mathrm{est}}$; (3) if $\hat\varepsilon$ is below the noise estimate and $L_{\mathrm{est}}<\eta L_{\min}$, either stop reducing the fit loss and increase a Jacobian penalty, or widen the hidden layer by 25--50%; (4) copy existing neurons and initialize new neurons with small output coefficients; (5) continue training. The theorem supplies the scaling law; $c_0,C_0$, the Jacobian estimator, and the widening threshold are empirical. First test on noisy two-moons, CIFAR-10 binary subsets, and synthetic spherical data using a width-swept two-layer ReLU baseline. Success is lower validation error or better adversarial/noise robustness at matched parameter-update FLOPs, together with fewer late-stage epochs spent driving training loss below the noise floor.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a two-layer ReLU noisy-regression MVP with the stated ReLU lower-bound controller, Jacobian probe estimator, and neuron-copy widening. The numerical sanity check clearly reproduced the claimed formula behavior: L_min decreased with width (0.2483 at m=2 to 0.0466 at m=64) and increased with fitting error (0.0207 at epsilon=0.08 to 0.1807 at epsilon=0.64). However, in the matched training experiment the residual only reached the known noise variance after the estimated Lipschitz constant had already exceeded 0.9 L_min, so widening never activated; baseline and controller were identical, including clean test MSE 0.04776. The proposed practical effect was therefore not observed.

### Mechanism check

- Verdict: Built a two-layer ReLU noisy-regression MVP with the stated ReLU lower-bound controller, Jacobian probe estimator, and neuron-copy widening. The numerical sanity check clearly reproduced the claimed formula behavior: L_min decreased with width (0.2483 at m=2 to 0.0466 at m=64) and increased with fitting error (0.0207 at epsilon=0.08 to 0.1807 at epsilon=0.64). However, in the matched training experiment the residual only reached the known noise variance after the estimated Lipschitz constant had already exceeded 0.9 L_min, so widening never activated; baseline and controller were identical, including clean test MSE 0.04776. The proposed practical effect was therefore not observed.
- Confidence: 9/10
- Limitations: Only a small synthetic noisy regression task was tested; no two-moons, CIFAR binary subset, repeated seeds, adversarial robustness, FLOP-normalized comparisons, or systematic calibration of c0/C0/eta was performed. The controller's empirical constants were not optimized, and the theorem's high-probability assumptions were not independently tested.

## Artifacts

- [experiment.py](https://synthcore.org/code/1386/experiment.py)
- [report.md](https://synthcore.org/code/1386/report.md)
- [results.json](https://synthcore.org/code/1386/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1386)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.