# Factor-Two Neural Model-Criticism Test

- ID: 1304
- Canonical URL: https://synthcore.org/idea/1304/factor-two-neural-model-criticism-test
- API JSON: https://synthcore.org/api/idea/1304.json
- API Markdown: https://synthcore.org/api/idea/1304.md
- Verification status: unverified
- Source: [arXiv:2607.23010](https://arxiv.org/abs/2607.23010)
- Category: theory
- Solves: accuracy, stability, sample-efficiency
- ML areas: vae, diffusion, world-model, loss, training-dynamics
- Math tags: probability, statistics, stochastic-processes
- Ratings: usefulness 5/10; difficulty 4/10; novelty 7/10

## Idea description

Use a frozen neural discrepancy score and conditional Monte Carlo replicas to test whether a generative model or learned sampler is compatible with a null data distribution, without requiring mixed chains or joint exchangeability. The resulting empirical p-value has a finite-sample false-alarm bound of at most two times the nominal level, making it safer than an ordinary Monte Carlo rank test for validation and deployment monitoring.

## Mathematical statement

The paper defines the empirical p-value p_m = (1 + sum over i of the indicator that T(X_i) is at least T(X)) divided by (m+1), where X is observed and X_1 through X_m, conditionally on X, are independent draws from a kernel P. The useful coupling condition is that, under the null, X and a copy X' are conditionally independent and identically distributed given a latent variable Z: X given Z follows Q(.-given-Z), X' given Z follows the same law, and X is conditionally independent of X' given Z. The main guarantee is P_H0(p_m less than or equal to alpha) less than or equal to 2 alpha for every finite m, every fixed statistic T, and without an MCMC mixing assumption. Here Q_0 is the null data law, P is the simulator or Markov transition kernel, Z is the latent coupling variable, T is a fixed large-values-are-bad discrepancy, m is the replica count, and alpha is the nominal test level. In a neural application, T can be a frozen classifier logit or encoder discrepancy, but it must not be fitted on the same evaluation examples used for the guarantee.

## Key formulas

- $$p_m=\frac{1+\sum_{i=1}^{m}{\mathbbm{1}}\left\{T(X_i)\geq T(X)\right\}}{m+1},\qquad X_1,\dots,X_m\mid X\stackrel{\mathrm{iid}}{\sim}P(\cdot\mid X).$$
- $$X\mid Z\sim Q(\cdot\mid Z),\qquad X'\mid Z\sim Q(\cdot\mid Z),\qquad X\perp X'\mid Z,$$
- $$\mathbb P_{H_0}\{p_m\leq\alpha\}\leq 2\alpha.$$
- $$T_\phi(x)=\ell_\phi(x)-\mathbb E_{y\sim Q_0}[\ell_\phi(y)]\quad\text{or}\quad T_\phi(x)=\|e_\phi(x)-\bar e_\phi\|_2^2.$$

## Implementation notes

Integrate this method at evaluation and monitoring time rather than inside the model's forward pass. First train a neural discrepancy T_phi on a disjoint split: for example, train a binary classifier between held-out real data and model-generated data, then freeze its logit ell_phi; alternatively use a frozen encoder and squared feature distance. For each held-out observation x, generate m replicas with the model or sampler kernel initialized from x: x_i is sampled from P(.|x). Compute t_0 = T_phi(x), t_i = T_phi(x_i), and p_m = (1 + number of i with t_i >= t_0)/(m+1). Pseudocode is: `t0 = T(x_batch); count = 0; for i in 1..m: xi = kernel(x_batch, independent_seed_i); count += (T(xi) >= t0); p = (1 + count)/(m + 1)`. The paper supplies the conservative interpretation: a threshold p_m <= alpha has false-alarm probability at most 2 alpha under the coupling assumptions. Use independent random seeds and independent chains; estimate replica correlations and effective sample size diagnostically, but do not claim validity if the simulator is not marginally null-compatible. The first experiment should use MNIST or CIFAR-10, a small VAE or diffusion sampler, a frozen classifier-based discrepancy, and an intentionally misspecified decoder. Compare this test with the ordinary Monte Carlo p-value and a fully independent-sample test for m equal to 7, 31, and 127. Success means the false-alarm rate remains below 2 alpha for a correctly specified model even with deliberately short, poorly mixed chains, while detection power increases on the misspecified model and the resulting early-stopping signal is more stable than raw validation loss.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.