Data-Processing-Safe Learnable Divergence

(1) Integrate the loss at a concrete stochastic channel. Start with knowledge distillation: let p_i=softmax(z_i^T/ au) be teacher probabilities and q_i=softmax(z_i^S/ au) be student probabilities over the same C classes. Also create an augmented teacher distribution p'_j by applying a known stochastic class-mixing or token-pooling channel K; use the divergence both for teacher-student matching and as a consistency term between two augmented views. Compute r_i=(p_i+\delta)/(q_i+\delta), and evaluate L_f=\sum_i(q_i+\delta)f_ heta(r_i), with \delta=10^{-6}. Add \lambda L_f to ordinary cross-entropy. (2) Pseudocode: initialize knot locations c_k on a log-spaced grid in [0,10]; learn unconstrained u_k and set w_k=softplus(u_k); for each batch, compute p and q, then r=(p+\delta)/(q+\delta), y_k=relu(r-c_k), f=epsilon*(r-1)^2+sum_k w_k*y_k^2, L=sum((q+delta)*f), and backpropagate total_loss=CE+lambda*L. Update the network and u_k with AdamW, while stopping gradients through p for a frozen teacher. (3) The mathematical guarantee is convexity from nonnegative weights and epsilon>0; the data-processing inequality is not estimated. Empirically estimate its usefulness by applying random row-stochastic K matrices or the actual augmentation channel and checking D_f(pK\|qK)\le D_f(p\|q) numerically. Monitor learned f curves, gradient norms, ratio clipping frequency, and whether the loss remains finite for peaked distributions. (4) First experiment: train a 6-layer CIFAR-10 or TinyImageNet student from a frozen wider teacher, comparing cross-entropy plus KL distillation against cross-entropy plus the proposed learnable divergence at equal parameter count, batch size, and training FLOPs. Run fixed f choices for KL, reverse-KL, and squared Hellinger as ablations, then enable learned knots. Success means lower student validation error or calibration error at the same training budget, improved robustness to strong augmentation, and empirical contraction under class-mixup or random stochastic channels. Failure is collapse of weights onto a numerically unstable tail, exploding ratios, or no improvement over KL after tuning lambda and temperature.