Distill a teacher's attention into a student by matching sink mass and the normalized content distribution as separate targets rather than applying one KL divergence to the entire attention row. Use the Aitchison distance on the content composition, which compares relative token allocation and prevents a large common sink probability from overwhelming differences between content tokens.
For models that predict probability distributions, replace the usual Wasserstein-2 loss or a Huber penalty on the final Wasserstein distance with a Huber penalty on quantile-by-quantile prediction errors. This suppresses gradients from localized outliers while retaining quadratic gradients on the majority of the distribution, which is useful for uncertainty prediction, histogram prediction, and distributional distillation.
Wrap a neural estimator with V leave-fold-out refits and use the dispersion of fold pseudo-values to produce uncertainty intervals without deriving an influence function or relying on unstable parameter-space Hessians. The same construction can be applied to scalar metrics, predictions at fixed inputs, dose-response curves, or vectors of logits and probabilities.
Use fresh unlabeled covariates to train a frozen-teacher student against pseudo-labels, then form an affine combination of teacher and student predictions. Estimate the combination weight on a small independent labeled calibration set, requiring no access to the teacher training data and no additional teacher or student fitting.