# Missingness-as-a-Label Signal

- ID: 79
- Canonical URL: https://synthcore.org/idea/79/missingness-as-a-label-signal
- API JSON: https://synthcore.org/api/idea/79.json
- API Markdown: https://synthcore.org/api/idea/79.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.22843](https://arxiv.org/abs/2608.22843)
- Category: training
- Solves: accuracy, sample-efficiency, generalization
- ML areas: loss, training, fine-tuning
- Math tags: probability, statistics, information-theory, optimization
- Ratings: usefulness 6/10; difficulty 4/10; novelty 5/10

## Idea description

Use the observed label-availability indicator as an auxiliary supervision signal when labels are preferentially missing for uncertain or difficult examples. Train the classifier with a joint likelihood containing both the class-label likelihood for labeled examples and a missingness likelihood whose probability depends on the classifier's posterior uncertainty.

## Mathematical statement

The paper models a latent class Z in {1,2} and an observed positive feature Y with class-conditional densities f_i(y;mu_i) = mu_i^{-1} exp(-y/mu_i), prior probabilities pi_i, and posterior tau_i(y;theta) = pi_i f_i(y;mu_i) / [pi_1 f_1(y;mu_1) + pi_2 f_2(y;mu_2)]. Its transferable construction is to make the missingness indicator M informative about this posterior. Here M=1 means the class label is missing and M=0 means it is observed. For a K-class neural classifier, p_theta(k|x) is the softmax posterior, H_theta(x) = -sum_{k=1}^K p_theta(k|x) log p_theta(k|x) is predictive entropy, and q_{theta,phi}(x) = Pr(M=1|x) = sigmoid(a + b H_theta(x)) is an entropy-logistic missingness model with nuisance parameters phi=(a,b). The per-example observed-data likelihood is [q_{theta,phi}(x)]^M [1-q_{theta,phi}(x)]^{1-M} times p_theta(z|x)^{1-M} for an observed label z. Thus unlabeled examples contribute log q_{theta,phi}(x), and labeled examples contribute log(1-q_{theta,phi}(x)) + log p_theta(z|x). The classifier receives gradients through H_theta, so the missingness pattern directly trains the decision boundary. A squared-discriminant variant replaces entropy by s_theta(x)^2, where s_theta(x) is a binary logit margin; this is useful when uncertainty is approximately symmetric around the decision boundary.

## Key formulas

- $$\tau_i(y;\bm{\theta})=\Pr(Z=i\mid Y=y)=\frac{\pi_i f_i(y;\mu_i)}{f_Y(y;\bm{\theta})},\qquad f_Y(y;\bm{\theta})=\pi_1f_1(y;\mu_1)+\pi_2f_2(y;\mu_2),$$
- $$H_\theta(x)=-\sum_{k=1}^{K}p_\theta(k\mid x)\log p_\theta(k\mid x),\qquad q_{\theta,\phi}(x)=\Pr(M=1\mid x)=\sigma\!\left(a+bH_\theta(x)\right),$$
- $$\mathcal{L}_{\mathrm{obs}}(x,m,z)=q_{\theta,\phi}(x)^m\,[1-q_{\theta,\phi}(x)]^{1-m}\,p_\theta(z\mid x)^{1-m},$$
- $$\mathcal{J}(\theta,\phi)=-\frac{1}{n}\sum_{j=1}^{n}\left[m_j\log q_{\theta,\phi}(x_j)+(1-m_j)\left(\log(1-q_{\theta,\phi}(x_j))+\log p_\theta(z_j\mid x_j)\right)\right]+\lambda(a^2+b^2).$$

## Implementation notes

Integrate this at the classifier loss, after the model produces logits for every training example. Let logits l_j=f_theta(x_j), probabilities p_j=softmax(l_j), and m_j=1 when z_j is unavailable. Compute entropy H_j=-sum_k p_{jk} log(p_{jk}+epsilon). Introduce two trainable scalar parameters a and b, and compute q_j=sigmoid(a+b*H_j). For each labeled item, use the ordinary cross-entropy plus the missingness term: loss_j=-log p_{j,z_j}-log(1-q_j). For each unlabeled item, use loss_j=-log q_j. Add lambda*(a^2+b^2), and optionally constrain b>=0 with b=softplus(beta) if the data-generation hypothesis is that uncertainty increases missingness. Compact pseudocode is: logits=model(x); p=softmax(logits); H=-(p*log(p+eps)).sum(-1); q=sigmoid(a+b*H); loss=((1-m)*CE(logits,z) - m*log(q+eps) - (1-m)*log(1-q+eps)).mean()+lambda*(a*a+b*b). Backpropagate through both p and H; do not detach the entropy in the first experiment, because the proposed information transfer comes from gradients through the posterior. The paper's exact mathematical ingredient is the posterior-based missingness likelihood; the neural approximation is the entropy-logistic link. Estimate a and b jointly, but also run an ablation fitting q from detached entropy to distinguish classifier improvement from mere missingness calibration. Start with a two-layer MLP on MNIST or CIFAR-10 using a fixed labeled/unlabeled split generated by q*=sigmoid(a*+b*H_true), where H_true is computed from a pretrained teacher or from class difficulty, and compare ordinary supervised cross-entropy, standard entropy-minimization, and the joint objective at equal numbers of observed labels. Measure test accuracy, calibration, negative log-likelihood, and performance as the labeled fraction changes. The expected signal is improved accuracy or NLL when missingness is genuinely uncertainty-dependent, with no gain or a possible degradation under random missingness; also verify that b recovers the sign of the synthetic mechanism and that the model does not simply drive all entropies to zero.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built the entropy-logistic missingness likelihood with trainable a and positive b, including a detached-entropy ablation and CPU/CUDA fallback. The math check matched dJ/da=mean(q-m) and dJ/db=mean((q-m)H) to numerical precision, with nonzero entropy-to-logit gradients. Across three seeds, joint training modestly improved informative-missingness test NLL but not accuracy; random missingness produced no meaningful gain.

### Mechanism check

- Verdict: Built the entropy-logistic missingness likelihood with trainable a and positive b, including a detached-entropy ablation and CPU/CUDA fallback. The math check matched dJ/da=mean(q-m) and dJ/db=mean((q-m)H) to numerical precision, with nonzero entropy-to-logit gradients. Across three seeds, joint training modestly improved informative-missingness test NLL but not accuracy; random missingness produced no meaningful gain.
- Confidence: 8/10
- Limitations: Only a small two-dimensional synthetic Gaussian binary task was tested; no MNIST/CIFAR, multiclass setting, label-fraction sweep, FLOP/speed study, or formal calibration metric was evaluated. The repeat experiment used three seeds and 500 full-batch optimization steps.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.79055
- Paired wins: 4/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/1168/bench_report.json)
- [bench_run_output.json](https://synthcore.org/code/1168/bench_run_output.json)
- [missingness_bench.py](https://synthcore.org/code/1168/missingness_bench.py)
- [missingness_signal.py](https://synthcore.org/code/1168/missingness_signal.py)
- [repeat_check.py](https://synthcore.org/code/1168/repeat_check.py)
- [repeat_results.json](https://synthcore.org/code/1168/repeat_results.json)
- [report.md](https://synthcore.org/code/1168/report.md)
- [report_bench_2026-09-03T010002.md](https://synthcore.org/code/1168/report_bench_2026-09-03T010002.md)
- [results.json](https://synthcore.org/code/1168/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1168)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.