Calibrated Prediction-Mixed Distillation

Integrate this as a post-training adaptation module around a frozen regression model or regression head. The teacher may be a neural network whose parameters are inaccessible; only a prediction API is required. First, sample a fresh unlabeled adaptation matrix Z from the deployment or target covariate distribution, query the teacher in batches to obtain t = f_T(Z), and train a small student model on the pairs (Z, t). For the first MVP, use a linear ridge student with beta_s = (Z^T Z / m + lambda_s I)^{-1} Z^T t / m, where m is the number of adaptation covariates, lambda_s is a tunable nonnegative regularization coefficient, and I is the p by p identity matrix. If the teacher exposes frozen embeddings, use those embeddings as Z and fit only a ridge output head. Next, reserve a separate labeled calibration set (X_c, y_c) that was not used for student fitting. Compute teacher predictions t_c = f_T(X_c), student predictions s_c = X_c beta_s, correction d_c = s_c - t_c, and teacher residual r_c = y_c - t_c. Set alpha_hat = clip_[0,1](d_c^T r_c / (d_c^T d_c + epsilon)). At inference, return f_T(x) + alpha_hat (s(x) - f_T(x)); alpha = 0 reproduces the teacher and alpha = 1 gives the pure student. The mathematical quantities computed exactly are the ridge solution and closed-form mixture coefficient; the target covariate distribution, residual covariance, and test risk are estimated empirically. Begin with California Housing or a synthetic anisotropic linear-regression task, deliberately use an over-regularized or distribution-shifted teacher, and compare the teacher, pure student, mixed predictor, and supervised fine-tuning using the same calibration-label budget. The primary success signal is lower target-domain MSE than both teacher and pure student, particularly when the unlabeled adaptation set is large and the labeled calibration set is small, with no teacher retraining and only one additional student evaluation at inference.