Smoothed Burg Proximal Optimizer

Integrate this at the optimizer level, initially only for tensors constrained to be positive or lie on a simplex: attention temperature or gating probabilities, MoE router probabilities, nonnegative factorization weights, or positive normalization parameters. Do not replace AdamW for all unconstrained Transformer weights in the first experiment. For a parameter vector $x$ with $x_i>0$, run the following loop: (1) compute the minibatch gradient $g=\nabla_xF(x)$; (2) evaluate $h_\varepsilon(x)=-\sum_i\log(x_i+\varepsilon)+(\lambda/2)\|x\|_2^2$; (3) approximately solve $x_{new}=\arg\min_{z>0}\{\langle g,z-x\rangle+R(z)+(1/\eta)D_{h_\varepsilon}(z,x)\}$; (4) renormalize if $x$ is a simplex variable; and (5) periodically update $\eta$ using a local relative-smoothness estimate. For $R=0$, solve each coordinate with 3-5 damped Newton iterations. The stationarity equation is $g_i+(1/\eta)[-1/(z_i+\varepsilon)+1/(x_i+\varepsilon)+\lambda(z_i-x_i)]=0$. Clamp $z_i+\varepsilon\geq10^{-8}$. For simplex routers, introduce a Lagrange multiplier for $\sum_i z_i=1$ and find it by bisection. Estimate local curvature from $q_t=[F(x_t)-F(x_{t-1})-\langle g_{t-1},x_t-x_{t-1}\rangle]/[D_h(x_t,x_{t-1})+10^{-8}]$, then use $\eta_t=0.5/(\max(q_t,10^{-8}))$. The paper provides the Bregman geometry and RRSC rationale; the Newton solver, empirical curvature estimator, and schedule are adaptations. First test a 2-layer MLP and a small Transformer on MNIST or CIFAR-10, applying the optimizer only to a softmax gate or positive attention-gate vector. Compare AdamW and ordinary entropic mirror descent at equal optimizer overhead. Measure loss versus steps and FLOPs, final accuracy, gradient variance, NaN frequency, dead coordinates, and probability collapse. Then test a 4-8 expert MoE on a toy language corpus, measuring perplexity, expert-load imbalance, and the fraction of router probabilities below $10^{-6}$.