ML: Distillation

Machine-learning ideas tagged Distillation in the ML taxonomy of the Math2NN corpus.

2 ideas found

ML: Distillation Math: Stochastic processes Clear all

Unverified 2026

Contractive Misspecification-Regularized State Model

Distill a large or accurate latent transition model into a smaller discrete-state recurrent model while penalizing both its one-step transition mismatch and its lack of contraction. The filtering perturbation bound predicts that reducing the Dobrushin coefficient prevents errors from accumulating over long sequences, while reducing the transition discrepancy lowers the irreducible steady-state error.

Useful6/10

Difficulty4/10

Novelty7/10

Regularization Stability Generalization Accuracy

Paper: An Operator-Theoretic Analysis of Nonlinear Filtering under Model Misspecification arXiv:2607.11378

✗ Mechanism failed 2026

KL-Budgeted Poisson-Race Sampling

Replace repeated rejection sampling from a neural proposal P with a Poisson race that selects one proposal candidate with exactly the target distribution Q. Candidate i receives a Poisson arrival time A_i and race score A_i divided by Q(X_i)/P(X_i); the smallest score wins. When Q is close to P in KL divergence, the selected index has low expected logarithmic rank, providing an adaptive compute budget instead of a fixed number of candidates.

Useful6/10

Difficulty5/10

Novelty6/10

Sampling Speedup Sample efficiency Accuracy

Paper: A note on shifting distributions via Poisson races arXiv:2608.26996