# Relevant-Noise RG Curriculum

- ID: 1721
- Canonical URL: https://synthcore.org/idea/1721/relevant-noise-rg-curriculum
- API JSON: https://synthcore.org/api/idea/1721.json
- API Markdown: https://synthcore.org/api/idea/1721.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.02716](https://arxiv.org/abs/2608.02716)
- Category: dynamics
- Solves: stability, generalization, sample-efficiency
- ML areas: training-dynamics, regularization, transformer, rnn
- Math tags: renormalization, dynamical-systems, probability, scaling
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Inject weak, unpostselected stochastic perturbations into activations, attention links, or recurrent transitions, but scale their strength according to effective computational size. The schedule is designed so that noise is initially a weak perturbation and becomes dominant only beyond a controlled depth or sequence length, producing a measurable crossover rather than uncalibrated constant dropout.

## Mathematical statement

Let g >= 0 be the stochastic perturbation strength, b > 1 a coarse-graining factor, and y_g > 0 its relevant RG eigenvalue. Under coarse graining, g' = b^(y_g) g. For effective size N, the crossover variable is x = g N^(y_g), and the crossover scale is N_* approximately equal to g^(-1/y_g). For N much smaller than N_*, the network follows approximately clean dynamics; for N much larger, stochasticity controls long-range behavior. A scale observable Q(N,g) should obey Q(N,g) = N^(-kappa) F(g N^(y_g)), where kappa is the clean-system scaling exponent and F is a crossover function. In a network, N may be layer depth, recurrent horizon, token distance, or optimizer time. The exponent y_g is estimated from data collapse across several noise strengths.

## Key formulas

- $$g' = b^{y_g} g, \qquad y_g > 0$$
- $$x(N,g) = g N^{y_g}, \qquad N_* \asymp g^{-1/y_g}$$
- $$Q(N,g) = N^{-\kappa} F\!\left(g N^{y_g}\right)$$
- $$h_{\ell+1} = \phi\!\left(W_\ell h_\ell + b_\ell + \sqrt{g_\ell}\,\xi_\ell\odot s_\ell\right), \qquad g_\ell = g_0\left(\frac{\ell}{\ell_0}\right)^{y_g}$$

## Implementation notes

1. Integration point: use a residual MLP, transformer, or RNN and inject zero-mean Gaussian noise after normalization and before the residual addition. Let the noise variance at layer or recurrent step l be g_l. For a transformer, apply the perturbation to the value stream or residual stream; for an RNN, apply it to the hidden-state transition. Use ordinary unpostselected training: every noisy forward pass contributes to the loss. 2. Pseudocode: choose g_0, a trial exponent y_g, a reference depth l_0, and a maximum variance g_max. At each layer, compute g_l = min(g_max, g_0 (l/l_0)^(y_g)), sample xi from a standard normal distribution, add sqrt(g_l) xi elementwise to the chosen activation, and backpropagate the averaged loss over one or two noise samples. 3. The RG scaling law and crossover form come from the paper; y_g, kappa, and F are estimated empirically. 4. First experiment: train 2-, 4-, and 8-layer residual MLPs on MNIST or CIFAR-10, comparing clean training, constant Gaussian noise, dropout, and the RG schedule. Measure activation correlations, gradient variance, and validation loss as functions of depth and g_0. The quantitative prediction is that curves for different g_0 collapse when plotted against x = g_0 N^(y_g), and that the largest change occurs near N_* = g_0^(-1/y_g). Fit y_g on half the settings and predict held-out crossover locations; agreement within 20 percent is required.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built a runnable Relevant-Noise RG experiment with a toy crossover verification and matched residual-MLP comparison. The toy checks confirmed RG rescaling to numerical precision, recovered y=0.685 versus 0.700, and measured crossover scales within 14.1% of N*=g^(-1/y), satisfying the 20% criterion; data-collapse scatter was strongly reduced. In the MLP, the schedule improved validation loss over clean, constant-noise, and dropout at depth 4 (0.0731 versus 0.0797, 0.1215, and 0.1038), but gains were not consistent at depths 2 and 8, so the practical training win is preliminary rather than universal.

### Mechanism check

- Verdict: Built a runnable Relevant-Noise RG experiment with a toy crossover verification and matched residual-MLP comparison. The toy checks confirmed RG rescaling to numerical precision, recovered y=0.685 versus 0.700, and measured crossover scales within 14.1% of N*=g^(-1/y), satisfying the 20% criterion; data-collapse scatter was strongly reduced. In the MLP, the schedule improved validation loss over clean, constant-noise, and dropout at depth 4 (0.0731 versus 0.0797, 0.1215, and 0.1038), but gains were not consistent at depths 2 and 8, so the practical training win is preliminary rather than universal.
- Confidence: 7/10
- Limitations: The toy observable is an explicitly constructed stochastic crossover model, so it validates the proposed scaling mechanism rather than deriving it from a physical network. The neural experiment used a small synthetic binary dataset, one seed, one noise hyperparameter, three depths, and 180 optimization steps; MNIST/CIFAR, broader hyperparameter sweeps, gradient-variance curves, multiple seeds, and a true held-out crossover fit were not tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.543
- Paired wins: 4/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/636/bench_report.json)
- [bench_rg_runner.py](https://synthcore.org/code/636/bench_rg_runner.py)
- [experiment.py](https://synthcore.org/code/636/experiment.py)
- [report.md](https://synthcore.org/code/636/report.md)
- [report_bench_2026-08-31T202001.md](https://synthcore.org/code/636/report_bench_2026-08-31T202001.md)
- [results.json](https://synthcore.org/code/636/results.json)
- [Download all files as ZIP](https://synthcore.org/download/636)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.