{
 "artifacts": null,
 "category": "optimization",
 "description": "Replace Euclidean momentum updates on simplex-valued neural variables with entropy mirror descent when the variable dimension is large and the loss is naturally smooth relative to negative entropy. The lower bound predicts that Nesterov-style acceleration cannot guarantee an asymptotic improvement in this geometry, while the geometry-matched update preserves positivity and can reduce boundary instability.",
 "formulas_latex": [
  "$$h(x)=\\sum_{k=1}^{d}x_k\\log(x_k),\\qquad \\Delta_d=\\{x\\in\\mathbb{R}^d:x_k\\ge 0,\\ \\sum_{k=1}^d x_k=1\\}.$$",
  "$$f(y)\\le f(x)+\\langle\\nabla f(x),y-x\\rangle+L D_h(y,x),\\qquad D_h(y,x)=h(y)-h(x)-\\langle\\nabla h(x),y-x\\rangle.$$",
  "$$\\\\inf_{\\\\text{first-order methods}}\\\\sup_{f\\\\ \\text{ relatively }L\\\\text{-smooth}}[f(x_T)-f(x^*)]=\\\\Omega\\\\left(\\\\frac{L}{T}\\\\right)\\\\quad\\\\text{when }d=\\\\Omega(T^2).$$",
  "$$x_{t+1,k}=\\frac{x_{t,k}\\exp(-\\eta g_{t,k})}{\\sum_{j=1}^{d}x_{t,j}\\exp(-\\eta g_{t,j})},\\qquad g_t=\\nabla_x\\ell(x_t),\\quad 0\u003c\\eta\\le 1/L.$$"
 ],
 "id": 1490,
 "implementation": "(1) Integrate the method into the optimizer update for a tensor whose final axis is a probability simplex: an MoE router probability vector, attention weights, learned mixture weights, or a categorical policy head. Do not apply it directly to ordinary unconstrained weights. Keep probabilities in float32, compute the loss gradient g_t with respect to the probability tensor x_t, and update each simplex independently. (2) Pseudocode: initialize x = softmax(z); at each minibatch compute loss and g = d loss / d x; compute u_k = log(x_k + epsilon) - eta*g_k; set x_new = softmax(u) over the simplex axis; optionally store z = u. The common additive shift in u is irrelevant. Use eta = 1/L when a bound is available. Otherwise start with eta = 0.1 and perform backtracking: evaluate the proposed step and increase L, equivalently decrease eta, until loss(x_new) \u003c= loss(x) + sum_k g_k*(x_new_k-x_k) + L*sum_k x_new_k*log((x_new_k+epsilon)/(x_k+epsilon)). (3) The mathematical contribution is the entropy Bregman geometry and the Omega(L/T) anti-acceleration lower bound. Estimate L empirically with minibatch backtracking; estimate D_h using the displayed KL expression and epsilon clamping. (4) First experiment: train a 64- or 256-expert Switch-style router on CIFAR-100 or a small language-model MoE. Compare AdamW on router logits, Nesterov-accelerated mirror descent, and entropy mirror descent with equal minibatches, gradient evaluations, and FLOPs. Measure validation loss, loss versus optimizer steps, routing entropy, fraction of dead experts, maximum gradient norm, and expert-load coefficient of variation. The falsifiable prediction is that acceleration will not improve loss scaling at large d and may be less stable, while entropy mirror descent will match or beat it in validation loss and produce fewer dead experts at equal compute.",
 "math_summary": "The feasible set is the probability simplex Delta_d = {x in R^d : x_k \u003e= 0 and sum_k x_k = 1}. The prox function is negative Shannon entropy h(x) = sum_k x_k log(x_k), with 0 log 0 = 0. Its Bregman divergence is D_h(y,x) = h(y) - h(x) - \u003cgrad h(x), y-x\u003e = sum_k y_k log(y_k/x_k), which is KL divergence for strictly positive probability vectors. Relative L-smoothness means f(y) \u003c= f(x) + \u003cgrad f(x), y-x\u003e + L D_h(y,x). The paper proves a worst-case first-order lower bound of Omega(L/T) for this class when d = Omega(T^2), so generic acceleration cannot guarantee the usual O(1/T^2) rate. Minimizing the linearized loss plus (1/eta)D_h(y,x_t), with g_t = grad f(x_t), produces exponentiated gradient: x_{t+1,k} = x_{t,k} exp(-eta g_{t,k}) / sum_j x_{t,j} exp(-eta g_{t,j}). Here x_t is the current simplex-valued neural variable, g_t is its loss gradient, eta is the step size, and L is the relative-smoothness constant.",
 "math_tags": [
  "convex-analysis",
  "optimization",
  "information-theory"
 ],
 "ml_areas": [
  "optimizer",
  "moe-routing",
  "attention"
 ],
 "paper": {
  "arxiv_id": "2607.27476",
  "arxiv_url": "https://arxiv.org/abs/2607.27476",
  "summary_what_math_gives_to_ml": "The paper proves that convex objectives relatively smooth with respect to negative Shannon entropy cannot generally obtain an accelerated O(1/T^2) rate: when the simplex dimension satisfies d=Omega(T^2), every first-order method has error at least Omega(L/T). The transferable asset is a principled warning against blindly adding Euclidean or Nesterov momentum to high-dimensional probability-valued neural variables. A practical experiment is to use entropy mirror descent for MoE router probabilities, attention distributions, or mixture weights and compare it with accelerated updates under matched gradient evaluations.",
  "title": "Entropy-Smooth Convex Optimization Cannot Be Accelerated",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 4,
  "usefulness": 6
 },
 "solves": [
  "stability",
  "sample-efficiency"
 ],
 "title": "Entropy-Geometry Anti-Acceleration Optimizer",
 "url": "https://synthcore.org/idea/1490/entropy-geometry-anti-acceleration-optimizer",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}