Dirichlet-calibrated mean-loss alarm
Implementation & benchmark of arXiv:2607.08415 — An Exact Distribution-Free Test for Means of Nonnegative Random Variables
Source paper: An Exact Distribution-Free Test for Means of Nonnegative Random Variables arXiv:2607.08415 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace heuristic moving-average thresholds for a nonnegative neural-network quantity with an exact finite-sample p-value computed from a batch of independent observations. Use the p-value to stop training, trigger a learning-rate reduction, or reject a model whose expected loss or safety cost exceeds a prescribed threshold, without assuming bounded, Gaussian, or identically distributed observations.
Formulas
Mathematical statement
For independent nonnegative random variables $X_1,\ldots,X_n$ and an independent vector $D=(D_0,D_1,\ldots,D_n)\sim\operatorname{Dir}(1,\ldots,1)$, define $K(x)=\mathbb{P}_D\{\sum_{i=1}^n x_iD_i\le1\}$, where $D_i\ge0$ and $\sum_{i=0}^nD_i=1$. The paper's main theorem states that if $\mathbb{E}X_i\le1$ for every $i$, then $\mathbb{P}\{K(X)\le\alpha\}\le\alpha$ for every $0\le\alpha\le1$; hence small $K(X)$ is a valid upper-tail p-value. For a neural-network quantity $Z$ with target mean threshold $\tau>0$, set $X_i=Z_i/\tau$, where $Z_i\ge0$ is evaluated on independent examples. Then $p=K(Z_1/\tau,\ldots,Z_n/\tau)$ is super-uniform when $\mathbb{E}Z\le\tau$. A practical Monte Carlo estimator samples $M$ independent Dirichlet vectors $D^{(m)}$ and computes $\widehat K=M^{-1}\sum_{m=1}^M\mathbf{1}\{\sum_iX_iD_i^{(m)}\le1\}$; use a conservative lower confidence bound on $K$ or increase $M$ before making an alarm decision.
Implementation notes
Integrate this at the training-monitoring layer, not inside the forward pass. Choose a nonnegative scalar $Z_i$ for each independently sampled example: per-example loss, squared error, energy, latency proxy, safety cost, or a nonnegative constraint violation. Set a target mean $\tau$ and collect a fresh validation batch $Z_1,\ldots,Z_n$ that is not reused for gradient updates. Normalize $X_i=Z_i/\tau$. To estimate $K$, sample $M$ Dirichlet vectors by drawing $G_j^{(m)}\sim\operatorname{Gamma}(1,1)$ and setting $D_j^{(m)}=G_j^{(m)}/\sum_{k=0}^{n}G_k^{(m)}$. Compute $S=\sum_m\mathbf{1}[\sum_{i=1}^nX_iD_i^{(m)}\le1]$ and use the one-sided Clopper–Pearson lower confidence bound $\operatorname{Beta}^{-1}(\delta;S,M-S+1)$ for $K$; this avoids falsely declaring a violation because of Monte Carlo noise. If the lower bound is at most alarm level $\alpha$, reduce the learning rate, increase regularization, or stop and reject the checkpoint. The theorem supplies the super-uniform guarantee; the engineer must empirically check independence, batch contamination, and Monte Carlo error. Start with a small MLP or ResNet on MNIST or CIFAR-10, using squared per-example loss and a target $\tau$ chosen from a baseline model. Compare this alarm with a moving-average mean threshold over repeated random seeds. Success means calibrated false-alarm frequency at or below $\alpha$ under a model known to satisfy the target, and earlier detection of mean-loss degradation under heavy-tailed or heteroscedastic validation losses at the same batch size.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.