On the Identifiability of Masked Prediction: Mode Blindness and Mask Schedules

What the math gives to ML

The paper identifies a failure mode of masked prediction that is directly relevant to masked language models and masked diffusion: schedules exposing almost the entire input can fit local conditionals while remaining exponentially insensitive to global mode probabilities. The transferable object is the schedule statistic \(\pi_s(\mu)\), which measures how much training mass uses at most \(s\) visible coordinates, together with positive probability on the fully masked input. A practical adaptation is to make mask visibility an explicit identifiability-control parameter rather than choosing it only for reconstruction quality. The most useful test is whether adding low-visibility or fully masked examples makes a model recover global mixture weights and improves unconditional generation without materially increasing conditional prediction loss.