Optimizer ideas

Optimizer update-rule variants and parameter-specific step sizes, benchmarked against a learning-rate-tuned baseline.

2 ideas found

Category: Optimizer ML: Regularization Math: Stochastic processes Clear all

Unverified 2026

Convex-mixture graph Langevin optimizer

Put a gradient-Gibbs prior on differences between connected neural parameters rather than on individual parameters, and evolve the parameters with Langevin steps generated from randomly selected strictly convex component energies. The aggregate regularizer may be non-convex, but every sampled component has controlled curvature and outward drift, providing a practical stability mechanism for noisy training.

Useful5/10

Difficulty5/10

Novelty6/10

Optimizer Stability Sample efficiency Generalization

Paper: Gradient Gibbs measures with non-convex potentials and the universality class of the Gaussian Free Field arXiv:2608.14526

Unverified 2026

Soft-potential collision optimizer

Maintain a small population of neural-network parameter replicas and interleave ordinary gradient steps with Boltzmann/Kac-style binary collisions. Each collision preserves the pair's mean parameter vector and relative-distance norm while randomly rotating the relative direction, with collision frequency proportional to a regularized negative power of replica distance.

Useful5/10

Difficulty6/10

Novelty7/10

Optimizer Sample efficiency Stability Optimizer

Paper: Quantitative propagation of chaos for the Boltzmann equation with moderately soft potentials arXiv:2607.13825