Optimizer ideas

Optimizer update-rule variants and parameter-specific step sizes, benchmarked against a learning-rate-tuned baseline.

2 ideas found

Category: Optimizer Solves: Accuracy Math: Statistics Clear all

✗ Failed on benchmark 2026

Martingale Response Control Variate

Use the trajectory martingale decomposition to separate predictable training updates from genuinely unpredictable residual updates, then scale the residual according to its estimated response to future loss. The method targets stochastic or event-driven optimization with history-dependent samples and predicts that response-weighted residual energy, rather than total gradient variance, controls update noise and instability.

Useful7/10

Difficulty7/10

Novelty7/10

Optimizer Stability Speedup Accuracy

Paper: The Memory Hidden in Response Fluctuations: Trajectory-Level Fluctuation-Response Theory and Inequalities for Non-Markovian Jump Dynamics arXiv:2608.20328

✗ Failed on benchmark 2026

Two-sided conditioned DFA

Replace the raw DFA outer-product update with a damped left-right preconditioned update that whitens both presynaptic activity directions and local-error directions. The activity factor removes nuisance-dominated input anisotropy, while the error factor equalizes postsynaptic credit coordinates; separate damping prevents noisy error covariances from destabilizing training.

Useful7/10

Difficulty5/10

Novelty5/10

Optimizer Accuracy Stability Sample efficiency

Paper: Conditioned Direct Feedback Alignment via Activity and Error Geometry arXiv:2607.18574