# Smooth-RG Modewise Optimizer

- ID: 979
- Canonical URL: https://synthcore.org/idea/979/smooth-rg-modewise-optimizer
- API JSON: https://synthcore.org/api/idea/979.json
- API Markdown: https://synthcore.org/api/idea/979.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.15784](https://arxiv.org/abs/2607.15784)
- Category: dynamics
- Solves: stability, speed, scalability
- ML areas: optimizer, training-dynamics, scheduler
- Math tags: dynamical-systems, renormalization, statistical-mechanics, spectral-theory, optimization
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Treat parameter-space curvature modes as RG momentum shells and use a smooth cutoff to construct a scale-dependent preconditioner rather than abruptly clipping eigenmodes. The optimizer should expose measurable crossovers between overdamped, KPZ-like, and nearly inviscid relaxation, allowing the learning rate and damping to change at empirically detected transitions instead of following a fixed schedule.

## Mathematical statement

The paper studies the Kuramoto–Sivashinsky equation $\partial_t h=\nu\partial_x^2h-\tau\partial_x^4h+(\lambda/2)(\partial_xh)^2$, with negative viscosity $\nu<0$ and stabilizing hyperviscosity $\tau>0$. Its functional RG has the exact flow $\partial_\kappa\Gamma_\kappa[\Phi]=(1/2)\operatorname{Tr}[(\Gamma_\kappa^{(2)}[\Phi]+R_\kappa)^{-1}\partial_\kappa R_\kappa]$, where $\kappa$ is the running inverse length scale, $\Phi$ denotes the fields, $\Gamma_\kappa$ is the effective action, $\Gamma_\kappa^{(2)}$ is its Hessian, and $R_\kappa$ is a smooth regulator. The NLO ansatz retains momentum- and frequency-dependent effective viscosity and noise: $\Gamma_\kappa=\int\bar\varphi\{\partial_t\varphi-f_\kappa^\nu(\partial_t,\partial_x)\partial_x^2\varphi-(\lambda/2)(\partial_x\varphi)^2-f_\kappa^D(\partial_t,\partial_x)\bar\varphi\}$. The linear response denominator is $i\omega+f_\kappa^\nu(\omega,p)p^2$, so define effective damping $a_\kappa(p)=f_\kappa^\nu(0,p)p^2$. For neural-network curvature modes, replace momentum $p$ by mode index $q$ and use a smooth regulator $R_s(q)=\alpha s^2/(\exp(q^2/s^2)-1)$, with spectral cutoff $s$ and damping coefficient $\alpha>0$. The predicted relaxation law is $t(q)\propto q^{-z}$ with $z\approx3/2$, $2$, and $1$ in the KPZ, Edwards–Wilkinson, and inviscid regimes respectively.

## Key formulas

- $$\partial_t h=\nu\partial_x^2h-\tau\partial_x^4h+\frac{\lambda}{2}(\partial_xh)^2,\qquad \nu<0,\ \tau>0.$$
- $$\partial_\kappa\Gamma_\kappa[\Phi]=\frac{1}{2}\operatorname{Tr}\left[(\Gamma_\kappa^{(2)}[\Phi]+R_\kappa)^{-1}\partial_\kappa R_\kappa\right].$$
- $$\Gamma_\kappa=\int_{t,x}\bar\varphi\left\{\partial_t\varphi-f_\kappa^\nu(\partial_t,\partial_x)\partial_x^2\varphi-\frac{\lambda}{2}(\partial_x\varphi)^2-f_\kappa^D(\partial_t,\partial_x)\bar\varphi\right\},\qquad a_\kappa(p)=f_\kappa^\nu(0,p)p^2.$$
- $$t(q)\propto q^{-z},\qquad z\approx\frac{3}{2}\ \text{(KPZ)},\quad z\approx2\ \text{(EW)},\quad z\approx1\ \text{(inviscid)}.$$

## Implementation notes

1. Integration point: use this as a replacement for AdamW on a small MLP or Transformer, partitioning parameters into approximate curvature modes. Every 100 optimization steps, run eight Hutchinson/Lanczos probes to estimate the largest 64 Hessian eigenvalues $q_i$ or block-curvature modes. Measure each mode's effective relaxation response by applying a small perturbation $\delta\theta_i$ and recording the projected gradient change over 5–20 optimizer steps. 2. Pseudocode: initialize $s$ above the largest measured mode; at each update estimate $a_i=\langle\Delta g_i,\delta\theta_i\rangle/\langle\delta\theta_i,\delta\theta_i\rangle$, compute $R_s(q_i)=\alpha s^2/[\exp(q_i^2/s^2)-1]$, and update $\theta_i\leftarrow\theta_i-\eta g_i/(|a_i|+R_s(q_i)+\epsilon)$. Decrease $s$ geometrically only after the current spectral shell has been sampled, using no hard eigenvalue cutoff. 3. Computed from the paper: the smooth-regulator construction, scale-shell interpretation, and expected exponents. Estimated empirically: Hessian modes, damping $a_i$, and crossover locations. Fit $\log t_i$ against $\log q_i$ in sliding windows to estimate $z$. 4. First cheap experiment: train a width-256 MLP on MNIST and a 2-layer Transformer on a character-level copy task, comparing AdamW, Shampoo, and this optimizer at matched compute. Measure gradient explosions, loss decay, and mode relaxation times. The falsifiable signature is a slope change near shell transitions: $z\approx2$ at intermediate modes, $z\approx1$ where estimated $a_i$ approaches zero, and $z\approx1.5$ after low-mode coarse-graining. Each crossover should remain within 20% under changes in batch size or width.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a smooth-regulator modewise optimizer and numerical verification suite. The regulator asymptotic predictions were confirmed, crossover roots matched numerical observations within 0.025%, and the predicted Euler stability boundary eta*=2.0 was confirmed. However, the relaxation exponents were z≈3.0 at low/intermediate modes and z≈0 at high modes, not the claimed z≈1.5, 2, or 1; the toy comparison also strongly favored the AdamW-like baseline, so the proposed mechanism was not validated.

### Mechanism check

- Verdict: Built a smooth-regulator modewise optimizer and numerical verification suite. The regulator asymptotic predictions were confirmed, crossover roots matched numerical observations within 0.025%, and the predicted Euler stability boundary eta*=2.0 was confirmed. However, the relaxation exponents were z≈3.0 at low/intermediate modes and z≈0 at high modes, not the claimed z≈1.5, 2, or 1; the toy comparison also strongly favored the AdamW-like baseline, so the proposed mechanism was not validated.
- Confidence: 9/10
- Limitations: Only a diagonal positive-quadratic toy problem was tested; no MLP, MNIST, Transformer, Hutchinson/Lanczos curvature estimation, adaptive shell schedule, noisy gradients, wall-clock comparison, or negative/indefinite curvature was evaluated. The claimed exponents were tested only as the implied relaxation law of the proposed diagonal update.

## Artifacts

- [report.md](https://synthcore.org/code/408/report.md)
- [results.json](https://synthcore.org/code/408/results.json)
- [smooth_rg_experiment.py](https://synthcore.org/code/408/smooth_rg_experiment.py)
- [smooth_rg_optimizer.py](https://synthcore.org/code/408/smooth_rg_optimizer.py)
- [Download all files as ZIP](https://synthcore.org/download/408)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.