Information-Complexity Transition Monitor

Integrate the monitor at a categorical neural-network interface: the softmax class distribution in a classifier, token distributions in a language model, attention weights, or gate probabilities in a mixture-of-experts layer. For each minibatch example b, compute probabilities p[b], clip them to a minimum of 1e−8, calculate H[b] = −sum(p[b] times log(p[b])), and calculate C[b] = sum(p[b] times (−log(p[b]) − H[b]) squared). Maintain an exponential moving average Cbar with smoothing coefficient alpha, and record its finite difference slope. Pseudocode is: p = softmax(logits / temperature); q = clamp(p, 1e−8, 1); H = −sum(q times log(q), last dimension); C = sum(q times (−log(q) − H[..., None]) squared, last dimension); Cbar = (1 − alpha) times Cbar plus alpha times mean(C); slope = Cbar − previous_Cbar. The information-variance formula comes from the paper; clipping, minibatch averaging, and exponential smoothing are empirical engineering choices. First run a small Transformer or MLP classifier on MNIST or CIFAR-10 while sweeping learning rate and softmax temperature. Compare ordinary training with a controller that halves the learning rate when C reaches a local maximum and its slope changes from positive to negative, and raises temperature when C collapses toward zero. The quantitative prediction is a nonmonotonic complexity curve: C should be low for saturated one-hot outputs and nearly uniform outputs, with an interior peak. A falsification test is whether the C peak predicts subsequent gradient-norm variance or validation-loss change better than entropy alone. A useful transition detector should identify this peak reproducibly across seeds and intervention should reduce post-peak collapse.