arXiv:2607.20213 2026 Architecture 2 ideas extracted · analyzed Aug 30, 2026
The paper provides a principled construction for converting metric geometry into weighted graphs: use a non-increasing radial kernel with dimension-dependent bandwidth normalization. This can transfer to attention and graph message passing as an explicit locality prior that is less sensitive to token density than unconstrained dot-product logits. A second transferable direction is regularization under small transport perturbations of empirical representation measures, encouraging predictions to remain stable when samples are rearranged or resampled. The strongest initial experiment is a manifold-kernel attention layer, with transport consistency as a separate training regularizer.
Replace or augment dot-product attention with a non-increasing radial kernel of pairwise representation distance. The bandwidth is normalized using an estimated local intrinsic dimension and local neighbor scale, creating an explicit locality-controlled attention operator.
Regularize a network so that its predictions remain stable when the empirical representation distribution is changed by a small transport perturbation. Couple two nearby augmented batches and penalize prediction disagreement under the coupling.