Factor-Two Neural Model-Criticism Test

Integrate this method at evaluation and monitoring time rather than inside the model's forward pass. First train a neural discrepancy T_phi on a disjoint split: for example, train a binary classifier between held-out real data and model-generated data, then freeze its logit ell_phi; alternatively use a frozen encoder and squared feature distance. For each held-out observation x, generate m replicas with the model or sampler kernel initialized from x: x_i is sampled from P(.|x). Compute t_0 = T_phi(x), t_i = T_phi(x_i), and p_m = (1 + number of i with t_i >= t_0)/(m+1). Pseudocode is: t0 = T(x_batch); count = 0; for i in 1..m: xi = kernel(x_batch, independent_seed_i); count += (T(xi) >= t0); p = (1 + count)/(m + 1). The paper supplies the conservative interpretation: a threshold p_m <= alpha has false-alarm probability at most 2 alpha under the coupling assumptions. Use independent random seeds and independent chains; estimate replica correlations and effective sample size diagnostically, but do not claim validity if the simulator is not marginally null-compatible. The first experiment should use MNIST or CIFAR-10, a small VAE or diffusion sampler, a frozen classifier-based discrepancy, and an intentionally misspecified decoder. Compare this test with the ordinary Monte Carlo p-value and a fully independent-sample test for m equal to 7, 31, and 127. Success means the false-alarm rate remains below 2 alpha for a correctly specified model even with deliberately short, poorly mixed chains, while detection power increases on the misspecified model and the resulting early-stopping signal is more stable than raw validation loss.