Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data
arXiv:2608.14712
2026
Regularization
2 ideas extracted · analyzed Sep 1, 2026
What the math gives to ML
The paper identifies a concrete failure mode in attention analysis: scalar metrics conflate the probability assigned to a sink token with the relative allocation among content tokens. Aitchison log-ratio geometry supplies an exact decomposition into a binary sink-mass channel and a composition-valued content channel, while entropy admits the corresponding decomposition into sink entropy plus conditional content entropy. This structure can transfer directly to attention distillation and head pruning, where preserving content routing may matter more than preserving sink behavior. The most promising implementation is to replace KL-only attention matching or aggregate head similarity with separately weighted sink and content objectives.
Ideas from this paper
✗ Mechanism failed
2026
Prune redundant attention heads using separate similarity scores for sink behavior and content routing. Two heads are considered safely redundant only when their normalized content compositions are close in Aitchison distance and their sink-mass trajectories are also close, avoiding pruning decisions dominated by a shared sink token.
Useful7/10
Difficulty4/10
Novelty7/10
✗ Mechanism failed
2026
Distill a teacher's attention into a student by matching sink mass and the normalized content distribution as separate targets rather than applying one KL divergence to the entire attention row. Use the Aitchison distance on the content composition, which compares relative token allocation and prevents a large common sink probability from overwhelming differences between content tokens.
Useful7/10
Difficulty3/10
Novelty7/10