Invariant-Measure Training Monitor

Integrate this first as an optimizer monitor rather than changing backpropagation. Use a deterministic full-batch two-layer MLP on MNIST or CIFAR-10, and define x_t as the concatenation of learning rate, log loss, gradient norm, parameter norm, and a fixed random projection of the parameter vector into 16 to 64 dimensions. Maintain a ring buffer of length 2W. After each optimizer step, append x_t; when the buffer is full, split it into windows A and B of length W. Estimate each occupation measure using fixed-grid histograms, random Fourier features, or a sliced Wasserstein distance. Compute observable averages for loss, accuracy, gradient norm, and parameter norm. Pseudocode is: update parameters; form x_t; append x_t; if the buffer is full, compute D(mu_A, mu_B), compute differences of all observable means, and set R equal to their weighted sum. Initially use R only for logging. In a second experiment, reduce the learning rate by a fixed factor whenever R exceeds a threshold for K consecutive windows. The paper supplies the occupation-measure construction; the embedding, finite-window estimator, and distance are empirical approximations. Sweep learning rates for SGD, momentum SGD, and Adam. The quantitative prediction is a regime boundary: below the largest stable learning rate, R and block-average variance should decay toward zero; above it, R should remain positive or show periodic peaks. Periodic dynamics should produce a finite autocorrelation period and a histogram with finitely many clusters, while chaotic dynamics may have a stable histogram but poor one-step state predictability. Repeat with W equal to 250, 500, and 1000; the estimated transition should vary by less than 20 percent.