# Smoothed Burg Proximal Optimizer

- ID: 441
- Canonical URL: https://synthcore.org/idea/441/smoothed-burg-proximal-optimizer
- API JSON: https://synthcore.org/api/idea/441.json
- API Markdown: https://synthcore.org/api/idea/441.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.05539](https://arxiv.org/abs/2607.05539)
- Category: optimization
- Solves: stability, speedup, accuracy
- ML areas: optimizer, moe-routing, attention
- Math tags: optimization, convex-analysis, geometry, information-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 5/10

## Idea description

Use a smoothed Burg entropy as the mirror map in a proximal-gradient optimizer for positive or simplex-valued neural parameters. The optimizer performs a Bregman-proximal step instead of an additive Euclidean update, while the smoothing parameter avoids the singularity of ordinary Burg entropy at zero.

## Mathematical statement

The paper studies convex composite minimization $\min_{\mathbf{x}\in\mathcal C}J(\mathbf{x})=F(\mathbf{x})+R(\mathbf{x})$, where $F$ is differentiable convex, $R$ is convex and possibly nonsmooth, and $\mathcal C$ is closed and convex. For a differentiable mirror map $h$, define the Bregman divergence $D_h(u,v)=h(u)-h(v)-\langle\nabla h(v),u-v\rangle$. The paper’s Restricted Relative Strong Convexity condition is a lower curvature bound measured in this divergence: $F(u)\geq F(v)+\langle\nabla F(v),u-v\rangle+\mu D_h(u,v)$ for relevant iterates and minimizers, with $\mu>0$. Use the smoothed Burg mirror map $h_\varepsilon(x)=-\sum_i\log(x_i+\varepsilon)+\frac{\lambda}{2}\|x\|_2^2$, where $\varepsilon>0$ prevents a singularity at zero and $\lambda\geq0$ adds Euclidean curvature. Its Bregman-proximal-gradient update is $x_{t+1}=\arg\min_{x\in\mathcal C}\{\langle\nabla F(x_t),x-x_t\rangle+R(x)+(1/\eta_t)D_{h_\varepsilon}(x,x_t)\}$. Under relative smoothness and a suitable step size, the paper establishes linear convergence; the engineering hypothesis is that smoothing supplies a usable curvature geometry where unsmoothed Burg entropy can fail.

## Key formulas

- $$\arg\min_{\mathbf{x}\in\mathcal C}J(\mathbf{x}),\qquad J(\mathbf{x}):=F(\mathbf{x})+R(\mathbf{x}).$$
- $$D_h(u,v)=h(u)-h(v)-\langle\nabla h(v),u-v\rangle.$$
- $$F(u)\geq F(v)+\langle\nabla F(v),u-v\rangle+\mu D_{h_\varepsilon}(u,v),\qquad h_\varepsilon(x)=-\sum_i\log(x_i+\varepsilon)+\frac{\lambda}{2}\|x\|_2^2.$$
- $$x_{t+1}=\arg\min_{x\in\mathcal C}\left\{\langle\nabla F(x_t),x-x_t\rangle+R(x)+\frac{1}{\eta_t}D_{h_\varepsilon}(x,x_t)\right\}.$$

## Implementation notes

Integrate this at the optimizer level, initially only for tensors constrained to be positive or lie on a simplex: attention temperature or gating probabilities, MoE router probabilities, nonnegative factorization weights, or positive normalization parameters. Do not replace AdamW for all unconstrained Transformer weights in the first experiment. For a parameter vector $x$ with $x_i>0$, run the following loop: (1) compute the minibatch gradient $g=\nabla_xF(x)$; (2) evaluate $h_\varepsilon(x)=-\sum_i\log(x_i+\varepsilon)+(\lambda/2)\|x\|_2^2$; (3) approximately solve $x_{new}=\arg\min_{z>0}\{\langle g,z-x\rangle+R(z)+(1/\eta)D_{h_\varepsilon}(z,x)\}$; (4) renormalize if $x$ is a simplex variable; and (5) periodically update $\eta$ using a local relative-smoothness estimate. For $R=0$, solve each coordinate with 3-5 damped Newton iterations. The stationarity equation is $g_i+(1/\eta)[-1/(z_i+\varepsilon)+1/(x_i+\varepsilon)+\lambda(z_i-x_i)]=0$. Clamp $z_i+\varepsilon\geq10^{-8}$. For simplex routers, introduce a Lagrange multiplier for $\sum_i z_i=1$ and find it by bisection. Estimate local curvature from $q_t=[F(x_t)-F(x_{t-1})-\langle g_{t-1},x_t-x_{t-1}\rangle]/[D_h(x_t,x_{t-1})+10^{-8}]$, then use $\eta_t=0.5/(\max(q_t,10^{-8}))$. The paper provides the Bregman geometry and RRSC rationale; the Newton solver, empirical curvature estimator, and schedule are adaptations. First test a 2-layer MLP and a small Transformer on MNIST or CIFAR-10, applying the optimizer only to a softmax gate or positive attention-gate vector. Compare AdamW and ordinary entropic mirror descent at equal optimizer overhead. Measure loss versus steps and FLOPs, final accuracy, gradient variance, NaN frequency, dead coordinates, and probability collapse. Then test a 4-8 expert MoE on a toy language corpus, measuring perplexity, expert-load imbalance, and the fraction of router probabilities below $10^{-6}$.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built a smoothed-Burg simplex proximal-gradient optimizer with exact coordinate inversion and Lagrange-multiplier bisection, plus Euclidean projected-gradient and entropic mirror-descent baselines. The mechanism checks succeeded: KKT residuals were below 7.3e-14, divergence stayed finite for every fixed epsilon, curvature scaled approximately as epsilon^-2 near zero, and empirical tangent contraction remained below one (0.940–0.969). In the toy quadratic, Burg was slower than both baselines, so this demonstrates the proposed geometry/stability mechanism but not a practical optimization win.

### Mechanism check

- Verdict: Built a smoothed-Burg simplex proximal-gradient optimizer with exact coordinate inversion and Lagrange-multiplier bisection, plus Euclidean projected-gradient and entropic mirror-descent baselines. The mechanism checks succeeded: KKT residuals were below 7.3e-14, divergence stayed finite for every fixed epsilon, curvature scaled approximately as epsilon^-2 near zero, and empirical tangent contraction remained below one (0.940–0.969). In the toy quadratic, Burg was slower than both baselines, so this demonstrates the proposed geometry/stability mechanism but not a practical optimization win.
- Confidence: 7/10
- Limitations: Only a small convex simplex quadratic was tested; no MLP, Transformer, MoE, MNIST/CIFAR data, adaptive curvature schedule, nonsmooth R, FLOP/speed measurements, or repeated statistical runs were evaluated. Finite divergence is guaranteed for fixed positive epsilon, while its magnitude grows sharply as epsilon approaches zero.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.59915
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_burg.py](https://synthcore.org/code/227/bench_burg.py)
- [bench_report.json](https://synthcore.org/code/227/bench_report.json)
- [burg_experiment.py](https://synthcore.org/code/227/burg_experiment.py)
- [report.md](https://synthcore.org/code/227/report.md)
- [report_bench_2026-08-30T211447.md](https://synthcore.org/code/227/report_bench_2026-08-30T211447.md)
- [results.json](https://synthcore.org/code/227/results.json)
- [verification.json](https://synthcore.org/code/227/verification.json)
- [verify.py](https://synthcore.org/code/227/verify.py)
- [Download all files as ZIP](https://synthcore.org/download/227)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.