On the Computational and Statistical Efficiency of the Empirical Maximum Entropy on the Mean Method

What the math gives to ML

The paper's transferable asset is the MEM function, which converts an empirical prior distribution into a convex regularizer on a mean or latent representation. It penalizes distributions over plausible states by their KL divergence while enforcing a prescribed mean, and its dual representation makes gradients computable using log-partition functions and softmax weights. A practical neural-network adaptation is to regularize an inverse-model output or latent code with an empirical MEM prior, using a small learned dual variable rather than a differentiable density estimator. The paper's O(n^{-1/2}) empirical stability result suggests that this regularizer can remain reliable when the unknown prior is replaced by a finite memory bank.