{
 "artifacts": [
  {
   "name": "experiment.py",
   "url": "https://synthcore.org/code/1033/experiment.py"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/1033/report.md"
  },
  {
   "name": "results.json",
   "url": "https://synthcore.org/code/1033/results.json"
  }
 ],
 "category": "optimization",
 "description": "Replace AdamW's single exponentially decaying second-moment accumulator with a small bank of accumulators whose combined impulse response approximates fractional relaxation. The resulting preconditioner remembers rare or old gradient directions with a power-law rather than geometric decay, which may improve optimization on nonstationary, sparse-gradient, or long-horizon problems.",
 "download_zip": "https://synthcore.org/download/1033",
 "formulas_latex": [
  "$$e_{\\alpha}(t):=E_{\\alpha}(-t^{\\alpha}):=\\sum_{n=0}^{\\infty}\\frac{(-t^{\\alpha})^{n}}{\\Gamma(\\alpha n+1)}$$",
  "$$e_{\\alpha}(t)\\sim \\frac{t^{-\\alpha}}{\\Gamma(1-\\alpha)},\\qquad t\\to\\infty,\\quad 0\u003c\\alpha\u003c1$$",
  "$$\\hat h_{\\ell}=\\sum_{k=1}^{K}w_k(1-\\rho_k)\\rho_k^{\\ell}\\approx h_{\\ell}:=E_{\\alpha}\\!\\left(-\\left(\\frac{\\ell+1}{\\tau}\\right)^{\\alpha}\\right)$$",
  "$$v_{t,k}=\\rho_kv_{t-1,k}+(1-\\rho_k)g_t^2,\\qquad \\tilde v_t=\\sum_{k=1}^{K}w_k\\frac{v_{t,k}}{1-\\rho_k^t},\\qquad \\theta_{t+1}=\\theta_t-\\eta\\frac{g_t}{\\sqrt{\\tilde v_t}+\\varepsilon}$$"
 ],
 "id": 2676,
 "implementation": "Integrate this at the AdamW second-moment line, keeping Adam's first-moment update and decoupled weight decay unchanged initially. Choose \\(K=4\\) or \\(K=8\\) decay factors logarithmically across the desired memory range, for example \\(\\rho_k=\\exp(-1/\\tau_k)\\) with \\(\\tau_k\\in\\{1,4,16,64,256,1024,4096,16384\\}\\). Precompute nonnegative weights \\(w_k\\) by least-squares fitting \\(\\hat h_\\ell=\\sum_k w_k(1-\\rho_k)\\rho_k^\\ell\\) to \\(h_\\ell=E_\\alpha(-((\\ell+1)/\\tau)^\\alpha)\\) on logarithmically spaced lags \\(\\ell\\in[0,L]\\), enforcing \\(w_k\\ge0\\) and \\(\\sum_kw_k=1\\). During training, maintain one tensor \\(v_k\\) per decay factor, update all of them from the elementwise squared gradient, apply bias correction \\(v_{t,k}/(1-\\rho_k^t)\\), and sum the corrected tensors using \\(w_k\\). Estimate no paper-specific quantity online except optional \\(\\alpha\\) and \\(\\tau\\), which can be selected on a validation set; begin with fixed \\(\\alpha\\in\\{0.3,0.5,0.7\\}\\). First test on a 6-layer Transformer trained on WikiText-103 and on CIFAR-10 with a small ResNet, comparing AdamW at equal learning-rate tuning budget and equal FLOPs. Also test sparse-gradient embedding parameters separately. Measure loss after a fixed number of optimizer steps, robustness to gradient accumulation and learning-rate changes, and validation perplexity. Success is faster loss descent or lower validation perplexity at equal steps without pathological growth in the oldest accumulator.",
 "math_summary": "The paper defines the Mittag-Leffler relaxation function for fractional order \\(\\alpha\\) as \\(e_{\\alpha}(t)=E_{\\alpha}(-t^{\\alpha})=\\sum_{n=0}^{\\infty}(-t^{\\alpha})^n/\\Gamma(\\alpha n+1)\\), where \\(t\\ge0\\), \\(\\Gamma\\) is the Gamma function, and \\(E_{\\alpha}\\) is the one-parameter Mittag-Leffler function. For \\(0\u003c\\alpha\u003c1\\), the paper states the long-time asymptotic \\(e_{\\alpha}(t)\\sim t^{-\\alpha}/\\Gamma(1-\\alpha)\\), so memory decays by a power law rather than exponentially. Approximate the target discrete kernel \\(h_\\ell=E_{\\alpha}(-((\\ell+1)/\\tau)^\\alpha)\\), with lag \\(\\ell\\ge0\\) and characteristic time \\(\\tau\u003e0\\), by a positive mixture of exponential kernels \\(\\hat h_\\ell=\\sum_{k=1}^{K}w_k(1-\\rho_k)\\rho_k^\\ell\\), where \\(w_k\\ge0\\), \\(\\sum_kw_k=1\\), and \\(0\u003c\\rho_k\u003c1\\). Each exponential is implemented by an EMA. The optimizer applies the weighted, bias-corrected mixture to elementwise squared gradients.",
 "math_tags": [
  "fractional-calculus",
  "dynamical-systems",
  "approximation-theory"
 ],
 "ml_areas": [
  "optimizer",
  "training-dynamics",
  "embedding"
 ],
 "paper": {
  "arxiv_id": "2608.21674",
  "arxiv_url": "https://arxiv.org/abs/2608.21674",
  "summary_what_math_gives_to_ml": "The paper identifies Mittag-Leffler relaxation as a constructive alternative to exponential forgetting: for fractional order \\(0\u003c\\alpha\u003c1\\), its tail decays as a power law \\(t^{-\\alpha}\\), retaining information over many timescales. This is transferable to neural systems because standard exponential moving averages, recurrent states, and linear state-space models have geometrically decaying memory and therefore forget old signals too quickly. The most practical adaptation is to approximate a Mittag-Leffler kernel with a small bank of exponential states, preserving long memory while retaining \\(O(KT)\\) computation and constant-size recurrent state. This can be tested both as a multi-timescale optimizer accumulator and as a long-context recurrent or SSM layer.",
  "title": "Anomalous diffusion memory factorization: Characteristic timescales and application to inverse problem",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 6,
  "usefulness": 7
 },
 "solves": [
  "stability",
  "sample-efficiency",
  "accuracy"
 ],
 "title": "Mittag-Leffler second-moment optimizer",
 "url": "https://synthcore.org/idea/2676/mittag-leffler-second-moment-optimizer",
 "verification": {
  "peer_reviewed": false,
  "stage1_mechanism_check": {
   "worked": false,
   "confidence": 9,
   "verdict": "Built a NumPy-only Mittag-Leffler EMA-bank optimizer prototype with simplex-constrained fitting and a deterministic toy benchmark. The target tail prediction was confirmed: observed log-log slope -0.4945 versus predicted -0.5. However, the stated normalized mixture could not approximate the target kernel (relative RMSE 0.951; weights collapsed to timescales 1 and 4), and the toy final loss was worse than AdamW, so the claimed long-memory mechanism was not realized.",
   "metrics": {
    "baseline": "AdamW toy final loss, seeds 0/1/2: 0.000545, 0.001002, 0.000733; mean 0.000760",
    "idea": "Mittag-Leffler bank toy final loss, seeds 0/1/2: 0.003656, 0.004101, 0.004674; mean 0.004143. Tail slope: predicted -0.5, observed -0.4945. Bank kernel relative RMSE 0.9509; fitted weights [0.1415, 0.8585, 0, 0, 0, 0]."
   },
   "how_to_run": "python3 experiment.py",
   "files": [
    "experiment.py",
    "results.json"
   ],
   "limitations": "Only alpha=0.5 was implemented because the compact fallback evaluates the half-order Mittag-Leffler function; no CIFAR-10, WikiText, Transformer, sparse embedding, hyperparameter tuning, or GPU experiment was run. The proposed kernel normalization itself appears inconsistent: the EMA mixture has unit total lag mass while the target relaxation values generally do not."
  },
  "status": "mechanism_failed",
  "status_label": "Mechanism failed",
  "updated_at": "2026-09-01T18:46:05",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}