# Rate-Controlled Smoothed-Sign Optimizer

- ID: 456
- Canonical URL: https://synthcore.org/idea/456/rate-controlled-smoothed-sign-optimizer
- API JSON: https://synthcore.org/api/idea/456.json
- API Markdown: https://synthcore.org/api/idea/456.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.06013](https://arxiv.org/abs/2607.06013)
- Category: optimization
- Solves: accuracy, stability, generalization
- ML areas: optimizer, training-dynamics, loss
- Math tags: optimization, convex-analysis, information-theory, dynamical-systems
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Replace the fixed numerical stabilizer in signSGD by an exponentially decaying stability path, so the optimizer remains sign-like for a controllable duration instead of eventually reverting toward ordinary gradient descent as gradients become small. Sweep the decay rate as an explicit implicit-bias parameter: slower annealing should retain the non-Euclidean, barrier-like bias, while faster annealing should approach the sign endpoint more closely.

## Mathematical statement

The paper studies the memoryless update w_{t+1}=w_t+eta_t[-nabla L(w_t)]/(|nabla L(w_t)|+epsilon_t), with division and absolute value coordinatewise and epsilon_t=epsilon_0 exp(-kappa t). Here w_t is the parameter vector, L is a weighted exponential loss, eta_t>0 is the step size, epsilon_0>0 is the initial stability constant, and kappa>=0 is the annealing rate. On separable linear data, the paper rewrites this recursion as entropic mirror ascent on a concave dual objective; the resulting dual-gap analysis uses a KL-divergence recursion and yields an S_t^{-1/2} normalized-iterate envelope, where S_t is cumulative optimization time. The static limiting geometry is a convex Burg-type barrier on a fixed margin slice. The neural-network adaptation preserves the exact coordinatewise smoothed-sign recursion and uses kappa to experimentally select how strongly this barrier-like implicit bias is retained.

## Key formulas

- $$w_{t+1}=w_t+\eta_t\frac{-\nabla L(w_t)}{|\nabla L(w_t)|+\epsilon_t},\qquad \epsilon_t=\epsilon_0\exp(-\kappa t)$$
- $$g_t=\nabla_{\theta}L(\theta_t),\qquad \theta_{t+1}=\theta_t-\eta_t\frac{g_t}{|g_t|+\epsilon_0e^{-\kappa t}}\quad\text{(coordinatewise)}$$
- $$L(w)=\sum_{i=1}^{n}q_i\exp(-a_i^\top w),\qquad a_i=y_i x_i,\quad q_i>0$$
- $$S_t=\sum_{s=0}^{t-1}\eta_s,\qquad \left\|\frac{w_t}{\|w_t\|}-\bar w\right\|\leq O(S_t^{-1/2})$$

## Implementation notes

(1) Integration point: implement this as a replacement for the optimizer update after backpropagation and before weight decay. Apply the fraction elementwise to every trainable tensor; initially exclude biases and normalization parameters, then test them in an ablation. Use decoupled weight decay separately, with p <- p - eta*lambda*p, so the measured effect is not confounded with L2 regularization. For minibatch training, use the current minibatch gradient, but first validate on full-batch or large-batch separable binary classification where the paper's assumptions are closest.

(2) Pseudocode:
```
initialize parameters theta, epsilon0, kappa
for t in 0,...,T-1:
    loss = weighted_exponential_or_logistic_loss(model(theta), batch)
    g = autograd(loss, theta)
    eps = epsilon0 * exp(-kappa * t)
    for parameter tensor p and gradient gp:
        p = p - eta[t] * gp / (abs(gp) + eps)
        p = p - eta[t] * weight_decay * p
```
The paper's mathematical ingredient is the exponential stability path and coordinatewise smoothed-sign map. The KL dual recursion and Burg-barrier characterization are used as motivation and diagnostics rather than assumed to hold for nonlinear networks. Log the normalized parameter direction theta divided by its Euclidean norm, training margin, coordinatewise ratio abs(g)/(abs(g)+eps), and cumulative time S_t=sum_s eta_s. Estimate direction stability using cosine similarity between checkpoints and compare the observed direction error against an empirical S_t^{-1/2} reference.

(3) First cheap experiment: train a linear classifier and a two-layer ReLU MLP on MNIST restricted to two classes, using logistic or exponential loss. Compare SGD, signSGD with fixed epsilon, Adam, and the annealed optimizer over a logarithmic grid of kappa values, with identical learning-rate tuning and matched optimizer steps. Start with width 256, batch sizes 256 and full batch. Success means a reproducible curve in margin versus test error as kappa varies, improved late-stage margin or robustness at equal training cost, and a smooth transition between fixed-epsilon gradient-like behavior and sign-like behavior. Falsify the idea if kappa has no effect after learning-rate retuning or if annealing is consistently worse than both tuned signSGD and Adam.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a reusable PyTorch RateControlledSmoothedSign optimizer and deterministic toy/classification experiment. The toy sweep confirmed all three mechanism predictions: for 0<kappa<1, asymptotic speed matched kappa within 1e-12 and the offset matched log((1-kappa)/(epsilon0*kappa)) within 2e-11; kappa>=1 saturated at sign speed 1. In the linear classification test, annealed variants reached 100% accuracy versus 98.8% for SGD/fixed smoothing, with the best normalized minimum margin 0.0476 at kappa=0.25, but this is only a small single-seed signal rather than a broad optimizer win.

### Mechanism check

- Verdict: Built a reusable PyTorch RateControlledSmoothedSign optimizer and deterministic toy/classification experiment. The toy sweep confirmed all three mechanism predictions: for 0<kappa<1, asymptotic speed matched kappa within 1e-12 and the offset matched log((1-kappa)/(epsilon0*kappa)) within 2e-11; kappa>=1 saturated at sign speed 1. In the linear classification test, annealed variants reached 100% accuracy versus 98.8% for SGD/fixed smoothing, with the best normalized minimum margin 0.0476 at kappa=0.25, but this is only a small single-seed signal rather than a broad optimizer win.
- Confidence: 9/10
- Limitations: Only a one-dimensional exponential-loss toy and one small full-batch linear logistic dataset were tested. No MLP, MNIST, minibatch noise, Adam comparison, hyperparameter retuning, multiple seeds, wall-clock benchmark, or robustness evaluation was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/230/bench_report.json)
- [experiment.py](https://synthcore.org/code/230/experiment.py)
- [rate_optimizer.py](https://synthcore.org/code/230/rate_optimizer.py)
- [report.md](https://synthcore.org/code/230/report.md)
- [report_bench_2026-08-30T211304.md](https://synthcore.org/code/230/report_bench_2026-08-30T211304.md)
- [results.json](https://synthcore.org/code/230/results.json)
- [stage2_bench.py](https://synthcore.org/code/230/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/230)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.