Unexplained Topology Distillation

1. Integration point: choose one or more intermediate feature tensors from a teacher and student, for example the spatial activation map after each residual block. For each channel, sample a small 2D or 3D cubical filtration from activation values and compute a persistence diagram containing birth-death points $u=(b,d)$; use persistence $d-b$ as the raw event weight. To keep the first prototype cheap, compute diagrams only for a random subset of batch examples and channels, or use a differentiable persistence implementation with gradients through point coordinates. 2. Pseudocode: given teacher features $F_T$ and student features $F_S$, construct diagrams $D_T,D_S$; set $w_i=\max(d_i-b_i,0)$ and $p_i=w_i/(\sum_k w_k+\varepsilon)$; for every teacher point compute $a_i=\max_j\exp(-\|u_i-v_j\|_2^2/(2\sigma^2))$; compute $L_{PCE}=-\sum_i p_i\log(p_i a_i+\varepsilon)$; optimize $L=L_{task}+\lambda(L_{PCE}-H(p_T))$. Stop gradients through the teacher diagram. 3. Computed from the paper’s mathematics: the event-space mismatch handling, induced probability, unexplained mass, and directional cross-entropy. Estimated empirically: persistence diagrams, the bandwidth $\sigma$, the teacher weights, and the coefficient $\lambda$. Estimate $\sigma$ as the median pairwise distance between teacher and student diagram points in the first training epoch, then optionally learn a positive per-layer bandwidth. Replace the nondifferentiable max with a temperature-controlled soft maximum, $a_i=\tau\log\sum_j\exp(s(u_i,v_j)/\tau)$ followed by clipping to $[0,1]$, if gradient stability is poor. 4. First experiment: distill a ResNet-50 teacher into a ResNet-18 student on CIFAR-100 or ImageNet-100, comparing ordinary logits plus feature-MSE distillation against the same setup with $L_{PCE}$ at one middle feature layer. Sweep $\lambda\in\{0.01,0.1,1\}$ and retain only diagrams with persistence above a small threshold. Measure top-1 accuracy, calibration, and accuracy under reduced-data training. A successful signal is higher student accuracy at equal training FLOPs, especially when the student has substantially fewer channels, together with decreasing teacher unexplained mass $p_T^S(\partial)$ and lower validation loss. An ablation should verify that reversing the direction to $S\to T$ or replacing PCE with a symmetric diagram distance removes or weakens the gain.