Adaptive Sliding-Mode Disturbance-Observer Optimizer

Integrate the method at the optimizer level, not inside individual neural-network layers. Maintain the parameter vector theta, velocity v, disturbance estimate d_hat, and adaptive gain k for each parameter group or for the entire model. At every minibatch, compute the ordinary gradient g. Form s = v + alpha times g, which avoids a Hessian-vector product and avoids finite-difference differentiation. Compute z = s divided by epsilon and use sat(z) = z divided by max(1, norm(z)) for groupwise vector saturation. Update d_hat by d_hat <- d_hat + eta_obs times ell times sat(z), then compute robust <- -d_hat - k times sat(z). Apply the discrete dynamics v <- beta v - eta g + eta robust and theta <- theta + v, with beta chosen near 1 - eta/tau. Update k by k <- max(k_min, k + eta times (gamma norm(s) - sigma(k-k0))). Clip d_hat and k only for numerical safety, while logging their unclipped values. The paper-derived mechanism is the observer, sliding surface, adaptive gain, and residual-dominance condition; Delta must be estimated empirically from the norm of observed minibatch or update mismatch. First test a two-layer MLP on MNIST, comparing SGD with momentum, Adam, and this optimizer at matched learning-rate budgets for batch sizes 32 and 512, with and without deliberately injected gradient noise. Measure the largest stable learning rate and log norm(s), k, d_hat, and update norms. The predicted signature is that divergence should disappear after the steady gain crosses the measured disturbance scale Delta. The median sliding error should decrease approximately as 1/k until reaching an epsilon-dependent floor. Estimate the empirical threshold k_c from the point where the slope of log(norm(s)) becomes negative; the prediction is k_c approximately equal to Delta within 20 percent. Increasing epsilon should increase the final error floor approximately proportionally, while increasing k above k_c should not significantly improve error below that floor.