# Late-Time Fractional-Order Optimizer

- ID: 350
- Canonical URL: https://synthcore.org/idea/350/late-time-fractional-order-optimizer
- API JSON: https://synthcore.org/api/idea/350.json
- API Markdown: https://synthcore.org/api/idea/350.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.01898](https://arxiv.org/abs/2607.01898)
- Category: optimization
- Solves: stability, sample-efficiency
- ML areas: optimizer, training-dynamics, scheduler
- Math tags: dynamical-systems, functional-analysis, special-functions
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Use the observed power-law decay of a scalar training signal to estimate the effective fractional order of the optimization dynamics, instead of choosing the memory exponent by hand. Then run a fractional-memory optimizer with the estimated order, allowing the algorithm to use stronger long-range memory during slow plateaus and weaker memory when the loss relaxes rapidly.

## Mathematical statement

The paper models a scalar observation of Caputo diffusion as $M_{\alpha}(t)=\sum_n a_nE_{\alpha,1}(-\lambda_nt^{\alpha})$, where $\alpha\in(0,1)$ is the Caputo order, $\lambda_n>0$ are eigenvalues, $a_n$ are signed observation coefficients, and $E_{\alpha,1}$ is the Mittag-Leffler function. After grouping equal eigenspaces, let $m$ be the first index with nonzero resolvent moment $S_m=\sum_n a_n/\lambda_n^m$. The late-time behavior is $M_{\alpha}(t)\propto t^{-m\alpha}$, and for two measurements at $t$ and $\rho t$, $\rho>1$, the paper gives $M_{\alpha}(\rho t)/M_{\alpha}(t)=\rho^{-m\alpha}(1+O(t^{-\alpha_0}))$, where $\alpha_0$ is a lower bound on the admissible fractional orders. Ignoring the asymptotic correction, this yields $\widehat{\alpha}=-\log(M(\rho t)/M(t))/(m\log\rho)$. The Caputo derivative is $\partial_t^\alpha u(t)=\Gamma(1-\alpha)^{-1}\int_0^t(t-s)^{-\alpha}u'(s)\,\mathrm ds$. For neural optimization, $M(t)$ is replaced by a positive smoothed loss or gradient norm, and $m=1$ is the default proxy because the true spectral moment is unavailable. The estimator is valid only after entering a monotone late-time regime and must be clipped to $[\alpha_{\min},\alpha_{\max}]$ while avoiding exceptional values satisfying $1/\Gamma(1-m\alpha)=0$.

## Key formulas

- $$\partial_{t}^{\alpha}u(t)=\frac{1}{\Gamma(1-\alpha)}\int_{0}^{t}(t-s)^{-\alpha}u^{\prime}(s)\,\mathrm{d}s$$
- $$M_{\alpha}(t)=\sum_{n}a_{n}E_{\alpha,1}(-\lambda_{n}t^{\alpha})$$
- $$\frac{M_{\alpha}(\rho t)}{M_{\alpha}(t)}=\rho^{-m\alpha}\left(1+O(t^{-\alpha_0})\right)$$
- $$\widehat{\alpha}_t=\operatorname{clip}_{[\alpha_{\min},\alpha_{\max}]}\left[-\frac{\log\!\left(\overline M(\rho t)/\overline M(t)\right)}{m\log\rho}\right]$$

## Implementation notes

(1) Integration point: add this as an AdamW replacement or wrapper around the parameter update. Maintain a short history of the scalar training signal $M_k$, preferably an exponential moving average of the minibatch loss above a positive floor, $M_k=\operatorname{EMA}(\ell_k-\ell_{\min}+\epsilon)$. Every $K$ optimizer steps compare the current signal with the signal approximately $K(\rho-1)$ steps earlier; use $m=1$, $\rho\in[1.5,2]$, and clip $\alpha$ to $[0.2,0.95]$. (2) Pseudocode: initialize $\theta$, gradient history $g_0=0$, a signal history, and $\alpha=0.5$. At step $k$, compute minibatch gradient $g_k$, update $M_k=\beta_M M_{k-1}+(1-\beta_M)(\ell_k-\ell_{\min}+\epsilon)$, and append it to a ring buffer. If $k\geq k_{\rm warmup}$ and the older value $M_{k-L}$ exists, calculate $r=M_k/M_{k-L}$, reject the estimate if $r\notin(0,1)$ or if the last $q$ ratios are not monotone, otherwise set $\alpha\leftarrow\operatorname{clip}[-\log r/\log\rho]$ with $\rho=k/(k-L)$ and exponentially smooth the estimate. Implement the fractional update using stored gradients: $v_k=\Gamma(\alpha)^{-1}\sum_{j=0}^{H-1}w_{k,j}(\alpha)g_{k-j}$, with piecewise-constant interval weights $w_{k,j}=((j+1)\Delta t)^\alpha-(j\Delta t)^\alpha$. Update $\theta_{k+1}=\theta_k-\eta v_k$ and optionally apply AdamW diagonal normalization to each stored gradient before accumulation. (3) Computed from the paper: the power-law ratio estimator and Caputo kernel exponent. Estimated empirically: the late-time window, loss floor, $m$, smoothing constants, and validity of monotonicity. (4) First experiment: train a small Transformer or ResNet on CIFAR-10 with equal token or update budgets, comparing AdamW, fixed-order fractional memory for $\alpha\in\{0.3,0.5,0.7,0.9\}$, and the adaptive method. Use history lengths $H\in\{32,128\}$ and measure validation accuracy, loss at equal FLOPs, gradient-norm variance, and sensitivity across random seeds. The method succeeds if it improves late-plateau loss or accuracy without destabilizing early training; useful signals are faster loss descent after warmup and lower variance than the best fixed-order baseline.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a late-time power-law order estimator and a piecewise-constant fractional-memory optimizer in PyTorch, with CUDA fallback and fixed-order controls. The synthetic math check recovered alpha=0.68 as 0.690 (absolute error 0.010), confirming the ratio formula. However, on the small quadratic task adaptive fractional memory reached tail loss 2.711 versus AdamW 0.00977, so the proposed optimizer showed no win and was substantially less stable in this test.

### Mechanism check

- Verdict: Built a late-time power-law order estimator and a piecewise-constant fractional-memory optimizer in PyTorch, with CUDA fallback and fixed-order controls. The synthetic math check recovered alpha=0.68 as 0.690 (absolute error 0.010), confirming the ratio formula. However, on the small quadratic task adaptive fractional memory reached tail loss 2.711 versus AdamW 0.00977, so the proposed optimizer showed no win and was substantially less stable in this test.
- Confidence: 9/10
- Limitations: Only a small full-batch quadratic regression was tested, not CIFAR-10, a neural network, minibatch noise, multiple random seeds, FLOP-matched training, gradient variance, or sensitivity studies. The fractional update uses normalized finite gradient history and does not implement AdamW diagonal normalization, so this is an MVP rather than a faithful production AdamW replacement.

## Artifacts

- [fractional_optimizer.py](https://synthcore.org/code/1272/fractional_optimizer.py)
- [report.md](https://synthcore.org/code/1272/report.md)
- [run_experiment.py](https://synthcore.org/code/1272/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/1272)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.