Bregman-Projection Polyak Optimizer

Integrate this optimizer at the parameter-update stage for a selected tensor, initially a simplex-valued MoE router, positive gate vector, or nonnegative factor in a low-rank layer. Keep the remaining network on AdamW so the first experiment isolates the geometry. For simplex probabilities use negative entropy $h(p)=\sum_i p_i\log p_i$, whose gradient is $\nabla h(p)=\log p+1$ and whose inverse produces $p_i(\lambda)\propto p_{t,i}\exp(-\lambda g_{t,i})$. For positive parameters use $h(w)=\sum_i(w_i\log w_i-w_i)$, giving $w_i(\lambda)=w_{t,i}\exp(-\lambda g_{t,i})$. Pseudocode: compute minibatch loss $f_t$ and gradient $g_t$; set $\delta=[f_t-\widehat f_\star]_+$; if $\delta=0$, skip the update; otherwise define $x(\lambda)=\nabla h^{-1}(\nabla h(x_t)-\lambda g_t)$; solve $\phi(\lambda)=\langle g_t,x_t-x(\lambda)\rangle-\delta=0$ by expanding an upper bracket and then performing 8--12 bisection iterations; update $x_{t+1}=x(\lambda)$ and renormalize simplex tensors. Stop gradients through the root solve. Use a known loss lower bound when available; otherwise maintain an exponential minimum of recent losses and subtract a margin, $\widehat f_\star=\min_{s\leq t}f_s-m$, with $m\in\{0.01,0.05,0.1\}$. The inverse mirror map is analytic, so the only numerical operation is a scalar root solve. First test on CIFAR-10 with a two-layer MLP containing a softmax router or positive gate. Compare against SGD, AdamW, and Euclidean Polyak with matched gradient evaluations. Record training loss versus steps and FLOPs, validation accuracy, gradient spikes, root-failure frequency, clipping frequency, and router entropy. A successful result is faster loss descent or lower final loss at equal updates without exploding entropy or requiring a learning-rate sweep.