# Unexplained Topology Distillation

- ID: 137
- Canonical URL: https://synthcore.org/idea/137/unexplained-topology-distillation
- API JSON: https://synthcore.org/api/idea/137.json
- API Markdown: https://synthcore.org/api/idea/137.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.24549](https://arxiv.org/abs/2608.24549)
- Category: regularization
- Solves: accuracy, generalization, sample-efficiency
- ML areas: distillation, loss, regularization, cnn
- Math tags: topology, probability, information-theory, metric-geometry
- Ratings: usefulness 6/10; difficulty 7/10; novelty 7/10

## Idea description

Add a directional persistent cross-entropy loss between teacher and student activation persistence diagrams. The loss assigns high probability to teacher topological events that the student reproduces, while accumulating the probability of unmatched teacher events in an explicit unexplained-event mass. This penalizes missing teacher structure without requiring teacher and student diagrams to have the same number of points.

## Mathematical statement

The paper starts from the ordinary cross-entropy expression $H(p_X,p_Y)=-\sum_{u\in D_X}p_X(u)\log p_Y(u)$, which is invalid when $p_X$ and $p_Y$ are defined on different persistence diagrams $D_X$ and $D_Y$. It instead constructs an induced probability $p_X^Y$ on $D_X\cup\{\partial\}$, where $\partial$ denotes an unexplained event, and evaluates $H(p_X,p_X^Y)=-\sum_{u\in D_X}p_X(u)\log p_X^Y(u)$. For an implementable neural adaptation, let $D_T=\{u_i\}_{i=1}^{n_T}$ and $D_S=\{v_j\}_{j=1}^{n_S}$ be teacher and student persistence diagrams, let $w_T(u_i)\ge 0$ be teacher persistence weights, and define $p_T(u_i)=w_T(u_i)/(\sum_k w_T(u_k)+\varepsilon)$. Let $s(u_i,v_j)\in[0,1]$ be a similarity kernel between diagram points, for example $s(u,v)=\exp(-\|u-v\|_2^2/(2\sigma^2))$. Define the explanation score $a_i=\max_j s(u_i,v_j)$. The induced student-explained probability is $p_T^S(u_i)=p_T(u_i)a_i$, while $p_T^S(\partial)=1-\sum_i p_T^S(u_i)$. Because $0\le a_i\le1$, this is a valid probability measure. The directional PCE loss is $\mathcal L_{\mathrm{PCE}}(T\to S)=-\sum_i p_T(u_i)\log(p_T(u_i)a_i+\varepsilon)$. Its excess over teacher entropy is $\mathcal L_{\mathrm{PCE}}-H(p_T)=-\sum_i p_T(u_i)\log(a_i+\varepsilon)$, which directly penalizes unexplained teacher events. The paper identifies the unexplained mass $p_T^S(\partial)$ with the total variation distance between the induced measure and the original teacher measure under this construction.

## Key formulas

- $$H(p_X,p_Y)=-\sum_{u\in D_X}p_X(u)\log p_Y(u)$$
- $$H(p_X,p_X^Y)=-\sum_{u\in D_X}p_X(u)\log p_X^Y(u)$$
- $$s(u,v)=\exp\!\left(-\frac{\|u-v\|_2^2}{2\sigma^2}\right),\quad a_i=\max_{1\le j\le n_S}s(u_i,v_j),\quad p_T^S(u_i)=p_T(u_i)a_i$$
- $$\mathcal L_{\mathrm{PCE}}(T\to S)=-\sum_{i=1}^{n_T}p_T(u_i)\log\bigl(p_T(u_i)a_i+\varepsilon\bigr),\quad p_T^S(\partial)=1-\sum_{i=1}^{n_T}p_T(u_i)a_i$$

## Implementation notes

1. Integration point: choose one or more intermediate feature tensors from a teacher and student, for example the spatial activation map after each residual block. For each channel, sample a small 2D or 3D cubical filtration from activation values and compute a persistence diagram containing birth-death points $u=(b,d)$; use persistence $d-b$ as the raw event weight. To keep the first prototype cheap, compute diagrams only for a random subset of batch examples and channels, or use a differentiable persistence implementation with gradients through point coordinates. 2. Pseudocode: given teacher features $F_T$ and student features $F_S$, construct diagrams $D_T,D_S$; set $w_i=\max(d_i-b_i,0)$ and $p_i=w_i/(\sum_k w_k+\varepsilon)$; for every teacher point compute $a_i=\max_j\exp(-\|u_i-v_j\|_2^2/(2\sigma^2))$; compute $L_{PCE}=-\sum_i p_i\log(p_i a_i+\varepsilon)$; optimize $L=L_{task}+\lambda(L_{PCE}-H(p_T))$. Stop gradients through the teacher diagram. 3. Computed from the paper’s mathematics: the event-space mismatch handling, induced probability, unexplained mass, and directional cross-entropy. Estimated empirically: persistence diagrams, the bandwidth $\sigma$, the teacher weights, and the coefficient $\lambda$. Estimate $\sigma$ as the median pairwise distance between teacher and student diagram points in the first training epoch, then optionally learn a positive per-layer bandwidth. Replace the nondifferentiable max with a temperature-controlled soft maximum, $a_i=\tau\log\sum_j\exp(s(u_i,v_j)/\tau)$ followed by clipping to $[0,1]$, if gradient stability is poor. 4. First experiment: distill a ResNet-50 teacher into a ResNet-18 student on CIFAR-100 or ImageNet-100, comparing ordinary logits plus feature-MSE distillation against the same setup with $L_{PCE}$ at one middle feature layer. Sweep $\lambda\in\{0.01,0.1,1\}$ and retain only diagrams with persistence above a small threshold. Measure top-1 accuracy, calibration, and accuracy under reduced-data training. A successful signal is higher student accuracy at equal training FLOPs, especially when the student has substantially fewer channels, together with decreasing teacher unexplained mass $p_T^S(\partial)$ and lower validation loss. An ablation should verify that reversing the direction to $S\to T$ or replacing PCE with a symmetric diagram distance removes or weakens the gain.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a compact directional PCE prototype for unequal-size persistence diagrams, including teacher persistence weighting, Gaussian explanation scores, induced unexplained-event mass, and optimization against a Chamfer control. The probability validity check passed and the excess identity error was 1.8e-7. In the main run unexplained mass fell from 0.488 to 0.147 with PCE versus 0.271 with Chamfer; across three additional starts PCE was better in two of three, giving a modest but promising signal rather than definitive evidence.

### Mechanism check

- Verdict: Built a compact directional PCE prototype for unequal-size persistence diagrams, including teacher persistence weighting, Gaussian explanation scores, induced unexplained-event mass, and optimization against a Chamfer control. The probability validity check passed and the excess identity error was 1.8e-7. In the main run unexplained mass fell from 0.488 to 0.147 with PCE versus 0.271 with Chamfer; across three additional starts PCE was better in two of three, giving a modest but promising signal rather than definitive evidence.
- Confidence: 6/10
- Limitations: This is a toy diagram-point optimization, not a CNN distillation experiment: no cubical-complex persistence computation, real activation tensors, classification task, differentiable persistence backend, bandwidth sweep, FLOP/accuracy measurement, calibration analysis, or reverse-direction ablation was tested. The control is a symmetric Chamfer diagram matching objective, and the repeated-start advantage was not universal.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.1876
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/1184/bench_report.json)
- [pce_experiment.py](https://synthcore.org/code/1184/pce_experiment.py)
- [report.md](https://synthcore.org/code/1184/report.md)
- [report_bench_2026-09-03T023821.md](https://synthcore.org/code/1184/report_bench_2026-09-03T023821.md)
- [results.json](https://synthcore.org/code/1184/results.json)
- [stage2_bench.py](https://synthcore.org/code/1184/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1184)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.