Modulated Scale-Residual Optimizer

Integrate this at the optimizer level for a scale-sensitive MLP or residual block. Choose one positive scalar a_l per selected layer, parameterized as a_l=softplus(q_l)+a_min, and maintain ordinary weights theta_l. Define a simple core parameterization theta_core,l(a_l)=a_l theta_ref,l, where theta_ref,l is initialized from the baseline model; the residual is v_l=theta_l-theta_core,l(a_l). At every minibatch, compute g_theta and g_q by backpropagation. Project the residual gradient away from the scale tangent t_l=partial theta_core,l/partial a_l using v_perp and t_l as in the final formula. Estimate rho_a over tensor coordinates i by rho_i=sqrt(a_l^2+(i/I_l-1/2)^2), and use w_i=rho_i^(-gamma), with 0<=gamma<1. Compute R_l=(mean_i[(w_i v_perp,i)^2])^(1/2). Add the modulation penalty lambda_mod sum_l R_l^2 to the loss, and add the extracted coupling proxy C_couple abs(da_l/dt) a_l^((n-3)/2+gamma) R_l ||v_l|| to the optimizer objective; C_couple is tuned on a pilot run. Update q_l with learning rate eta_a and residual weights with eta_v, choosing residual damping kappa so kappa >= (K-1)(2-gamma)/2 for a chosen schedule exponent K, while eta_a is at least 4x smaller. Pseudocode: forward; compute loss plus penalties; backward; project g_theta; theta_core=theta_core-eta_v g_residual; q=q-eta_a g_q; clip R_l to its running 95th percentile. The paper supplies the weighted coupling structure and rate condition; rho_a, projection, and constants are explicit engineering adaptations. First test on CIFAR-10 with a 6-layer residual MLP and compare AdamW, layer-scale residual initialization, and this optimizer at equal FLOPs. Track loss versus steps, gradient-norm spikes, R_l, and validation accuracy. Success means fewer unstable runs, at least 10% faster loss descent, or improved accuracy at matched training compute.