Transport-stable representation regularization
Implementation & benchmark of arXiv:2607.20213 — Graphon as a Bridge between Graphs and Manifolds
Source paper: Graphon as a Bridge between Graphs and Manifolds arXiv:2607.20213 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Regularize a network so that its predictions remain stable when the empirical representation distribution is changed by a small transport perturbation. Couple two nearby augmented batches and penalize prediction disagreement under the coupling.
Formulas
Mathematical statement
The paper defines the empirical measure mu_n=(1/n) sum from i=1 to n of delta_{x_i}, where delta_{x_i} is the Dirac measure at x_i. A sequence is admissible if there are transport maps T_n from the population measure mu to mu_n such that T_n#mu=mu_n and ||Id-T_n||_infinity tends to zero. Here T_n#mu is the pushforward measure and the infinity norm measures the maximum displacement. The paper states that independently sampled points are admissible almost surely. We use the finite-sample implication that two nearby empirical measures should produce similar predictions after matching their points. For embeddings z_i and z'_j, define a coupling pi with uniform row and column marginals and penalize the coupled prediction distance sum over i,j of pi_i_j times ||f(z_i)-f(z'_j)||_2^2. The coupling can be obtained with entropic optimal transport. The transport construction is motivated by the paper; Sinkhorn iterations are the concrete numerical approximation used by the implementation.
Implementation notes
Add this loss to supervised classification or contrastive training. For every minibatch, generate a second nearby sample set using weak augmentation, small Gaussian perturbations, token dropout, or mild feature masking. Run both versions through the encoder to obtain embeddings Z and Z-prime and logits P and P-prime. Compute the cost matrix C_i_j=||stop-gradient(z_i)-stop-gradient(z-prime_j)||_2^2. Obtain a coupling with entropic Sinkhorn iterations: initialize u and v as positive vectors; set K_i_j=exp(-C_i_j/tau); repeatedly update u_i=(1/n)/(sum_j K_i_j v_j) and v_j=(1/n)/(sum_i K_i_j u_i). Form pi_i_j=u_i K_i_j v_j. Stop gradients through pi initially. Add lambda times sum over i,j of pi_i_j ||P_i-P-prime_j||_2^2 to the ordinary cross-entropy loss, ramping lambda from zero during the first 10 percent of training. The empirical-measure and vanishing-transport-displacement motivation comes from the paper; the entropic coupling and Sinkhorn solver are the implementation choices. Monitor the average transport cost sum_i,j pi_i_j C_i_j and disable the regularizer for unusually large costs, so unrelated examples are not forced together. First test on CIFAR-10 with a four-layer CNN or ViT-Tiny against standard consistency regularization using the same augmentations. Measure clean accuracy, corruption accuracy, calibration, prediction variance under resampling, and training overhead. Success is improved robustness or calibration at matched clean accuracy, with less than 20 percent additional training time.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.