# Modulated Scale-Residual Optimizer

- ID: 387
- Canonical URL: https://synthcore.org/idea/387/modulated-scale-residual-optimizer
- API JSON: https://synthcore.org/api/idea/387.json
- API Markdown: https://synthcore.org/api/idea/387.md
- Verification status: unverified
- Source: [arXiv:2607.03152](https://arxiv.org/abs/2607.03152)
- Category: dynamics
- Solves: stability, speedup, generalization
- ML areas: optimizer, training-dynamics, mlp
- Math tags: dynamical-systems, spectral-theory, geometry, pde
- Ratings: usefulness 5/10; difficulty 5/10; novelty 7/10

## Idea description

Split the trainable state into an explicit scalar scale coordinate and a residual perturbation, then update them with separate time scales. Penalize residuals according to their distance from the scale-dependent core, so the optimizer cannot obtain apparent progress by destabilizing the scale mode. The method is a neural optimization analogue of the paper's modulation argument, not a direct consequence of the geometric singularity theorem.

## Mathematical statement

The paper controls a perturbation v around a scale-dependent profile indexed by a>0, with radial quantity rho_a(s) satisfying rho_a(s) approximately sqrt(a^2+s^2). The extracted estimate bounds a scale-variation coupling by |a'| a^((n-3)/2+gamma) ||v|| sup_{|s|<=s_0} rho_a(s)^(-gamma)|v(s,tau)|, where n is the ambient dimension parameter, gamma is a weight exponent, s is the spatial/core coordinate, a' is da/dtau, and ||v|| is an L2-type norm. It also uses the weighted perturbation quantity sup_tau a^(-2+gamma)||v||_{2,alpha;(-s_0,s_0)}^(gamma) and requires a spectral decay rate kappa satisfying (K-1)(2-gamma)/2 <= kappa, where K is the targeted singularity exponent and alpha is a Holder regularity index. The ML adaptation replaces s by layer or feature coordinates, a by an explicit positive network scale, and v by the residual after projecting parameters onto the scale tangent; the weighted norm supplies a computable trust-region penalty and the rate inequality supplies a conservative residual damping schedule.

## Key formulas

- $$\left|\frac{a^{\prime}}{2}\left\langle v,v\frac{\partial_{a}d\mu_{a}}{d\mu_{a}}\right\rangle\right|\leq C|a^{\prime}|a^{\frac{n-3}{2}+\gamma}\|v\|\left(\sup_{|s|\leq s_{0}}\rho_{a}^{-\gamma}|v|\right),$$
- $$\sup_{\tau}a^{-2+\gamma}\|v(\cdot,\tau)\|_{2,\alpha;(-s_{0},s_{0})}^{(\gamma)}\leq C_{0}^{2-\gamma}\epsilon+C\delta,$$
- $$\frac{K-1}{2}(2-\gamma)\leq\kappa,$$
- $$\rho_a(s)=\sqrt{a^2+s^2},\qquad \theta=\theta_{\mathrm{core}}(a)+v,\qquad v_{\perp}=v-\frac{\langle v,\partial_a\theta_{\mathrm{core}}(a)\rangle}{\|\partial_a\theta_{\mathrm{core}}(a)\|^2}\partial_a\theta_{\mathrm{core}}(a).$$

## Implementation notes

Integrate this at the optimizer level for a scale-sensitive MLP or residual block. Choose one positive scalar a_l per selected layer, parameterized as a_l=softplus(q_l)+a_min, and maintain ordinary weights theta_l. Define a simple core parameterization theta_core,l(a_l)=a_l theta_ref,l, where theta_ref,l is initialized from the baseline model; the residual is v_l=theta_l-theta_core,l(a_l). At every minibatch, compute g_theta and g_q by backpropagation. Project the residual gradient away from the scale tangent t_l=partial theta_core,l/partial a_l using v_perp and t_l as in the final formula. Estimate rho_a over tensor coordinates i by rho_i=sqrt(a_l^2+(i/I_l-1/2)^2), and use w_i=rho_i^(-gamma), with 0<=gamma<1. Compute R_l=(mean_i[(w_i v_perp,i)^2])^(1/2). Add the modulation penalty lambda_mod sum_l R_l^2 to the loss, and add the extracted coupling proxy C_couple abs(da_l/dt) a_l^((n-3)/2+gamma) R_l ||v_l|| to the optimizer objective; C_couple is tuned on a pilot run. Update q_l with learning rate eta_a and residual weights with eta_v, choosing residual damping kappa so kappa >= (K-1)(2-gamma)/2 for a chosen schedule exponent K, while eta_a is at least 4x smaller. Pseudocode: forward; compute loss plus penalties; backward; project g_theta; theta_core=theta_core-eta_v g_residual; q=q-eta_a g_q; clip R_l to its running 95th percentile. The paper supplies the weighted coupling structure and rate condition; rho_a, projection, and constants are explicit engineering adaptations. First test on CIFAR-10 with a 6-layer residual MLP and compare AdamW, layer-scale residual initialization, and this optimizer at equal FLOPs. Track loss versus steps, gradient-norm spikes, R_l, and validation accuracy. Success means fewer unstable runs, at least 10% faster loss descent, or improved accuracy at matched training compute.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.