Research ideas

Critical-Rate Learning-Rate Controller

Replace a fixed or manually scheduled learning rate with a feedback controller that estimates the critical rate of a saddle-node-like training mode and slows the schedule before the mode overshoots. The controller is applied to a low-dimensional observable of training, while ordinary gradient updates remain unchanged. It should permit aggressive learning-rate increases away from the bifurcation and automatically reduce them near a sharp stability boundary.