Optimizer ideas

Optimizer update-rule variants and parameter-specific step sizes, benchmarked against a learning-rate-tuned baseline.

2 ideas found

Category: Optimizer ML: Training dynamics Math: Statistics Clear all

✗ Failed on benchmark 2026

Martingale Response Control Variate

Use the trajectory martingale decomposition to separate predictable training updates from genuinely unpredictable residual updates, then scale the residual according to its estimated response to future loss. The method targets stochastic or event-driven optimization with history-dependent samples and predicts that response-weighted residual energy, rather than total gradient variance, controls update noise and instability.

Useful7/10

Difficulty7/10

Novelty7/10

Optimizer Stability Speedup Accuracy

Paper: The Memory Hidden in Response Fluctuations: Trajectory-Level Fluctuation-Response Theory and Inequalities for Non-Markovian Jump Dynamics arXiv:2608.20328

✗ Failed on benchmark 2026

Projection-Regularized Gradient Updates

Replace unconstrained neural-network updates by updates projected toward directions supported by a recent, regularized gradient or feature subspace. This transfers PRPC's errors-in-variables correction: directions that are weakly identified by noisy or rank-deficient minibatches receive stronger shrinkage, preventing large updates caused by accidental correlations. The method is especially suitable for recurrent, world-model, and small-data fine-tuning problems where minibatch covariance is…

Useful7/10

Difficulty6/10

Novelty6/10

Optimizer Stability Generalization Speedup

Paper: Projection-Regularized Indirect Data-Driven Predictive Control arXiv:2607.28123