Measure-Valued Forecast Martingale Regularizer

(1) Integration point: expose a predictive distribution at every information-update step of an RNN, SSM, iterative classifier, or diffusion denoiser. For a categorical target, each head produces logits \(z_i\) and probabilities \(p_i=\operatorname{softmax}(z_i)\). Add the regularizer to cross-entropy or negative log-likelihood during training; do not force adjacent forecasts to be equal, since legitimate information can change the distribution. (2) Pseudocode: for each minibatch of size \(B\), compute \(p_1,\ldots,p_n\). For each \(i<n\), construct \(H_i=[p_1;\ldots;p_i]\), evaluate bounded history features \(h_r(H_i)\) for \(r=1,\ldots,R\), and calculate \(m_{ikr}=B^{-1}\sum_{b=1}^{B}(p^{(b)}_{i+1,k}-p^{(b)}_{i,k})h_r(H_i^{(b)})\). Use \(L=L_{\mathrm{NLL}}+\lambda\sum_{i,k,r}m_{ikr}^2\), then backpropagate. Use coordinate features, random Fourier features, or a small frozen/randomly refreshed feature MLP; standardize each feature over the batch. (3) The paper supplies the martingale condition; finite-batch moments and the selected feature family are approximations. Evaluate held-out moment norms and bootstrap confidence intervals, and include \(h=1\) plus nonlinear history features so the test is not merely unconditional mean preservation. (4) First experiment: train a 6-step GRU classifier on sequentially revealed MNIST pixels or CIFAR-10 patches. Compare cross-entropy alone with \(\lambda\in\{0.01,0.1,1\}\). Measure accuracy, NLL, expected calibration error at every step, held-out martingale moment norm, and robustness under corrupted or shifted inputs. Success means lower held-out moment violation and calibration error without accuracy degradation, with under 10% additional training FLOPs.