On Transformer Dynamics

What the math gives to ML

The paper recasts transformer token interactions as directed two-body forces on a manifold and identifies a combinatorial route to representing arbitrary nonlocal, nonreciprocal attention patterns. The transferable asset is the decomposition of a directed attention graph into a small number of complete bipartite subgraphs, equivalently introducing shared hubs that mediate many source-target interactions. This suggests replacing an O(N^2) attention matrix by a learned hub-factorized attention operator whose cost is O(NK), where K is the number of selected bicliques or hubs. The most credible first use is structured sparse or low-rank attention, with the biclique cover computed from a pilot attention graph and then fine-tuned jointly.