Rate-Controlled Smoothed-Sign Optimizer

(1) Integration point: implement this as a replacement for the optimizer update after backpropagation and before weight decay. Apply the fraction elementwise to every trainable tensor; initially exclude biases and normalization parameters, then test them in an ablation. Use decoupled weight decay separately, with p <- p - eta*lambda*p, so the measured effect is not confounded with L2 regularization. For minibatch training, use the current minibatch gradient, but first validate on full-batch or large-batch separable binary classification where the paper's assumptions are closest.

(2) Pseudocode:

initialize parameters theta, epsilon0, kappa
for t in 0,...,T-1:
    loss = weighted_exponential_or_logistic_loss(model(theta), batch)
    g = autograd(loss, theta)
    eps = epsilon0 * exp(-kappa * t)
    for parameter tensor p and gradient gp:
        p = p - eta[t] * gp / (abs(gp) + eps)
        p = p - eta[t] * weight_decay * p

The paper's mathematical ingredient is the exponential stability path and coordinatewise smoothed-sign map. The KL dual recursion and Burg-barrier characterization are used as motivation and diagnostics rather than assumed to hold for nonlinear networks. Log the normalized parameter direction theta divided by its Euclidean norm, training margin, coordinatewise ratio abs(g)/(abs(g)+eps), and cumulative time S_t=sum_s eta_s. Estimate direction stability using cosine similarity between checkpoints and compare the observed direction error against an empirical S_t^{-1/2} reference.

(3) First cheap experiment: train a linear classifier and a two-layer ReLU MLP on MNIST restricted to two classes, using logistic or exponential loss. Compare SGD, signSGD with fixed epsilon, Adam, and the annealed optimizer over a logarithmic grid of kappa values, with identical learning-rate tuning and matched optimizer steps. Start with width 256, batch sizes 256 and full batch. Success means a reproducible curve in margin versus test error as kappa varies, improved late-stage margin or robustness at equal training cost, and a smooth transition between fixed-epsilon gradient-like behavior and sign-like behavior. Falsify the idea if kappa has no effect after learning-rate retuning or if annealing is consistently worse than both tuned signSGD and Adam.