Mechanism failed
2026
Channel-aware attention-head pruning
Prune redundant attention heads using separate similarity scores for sink behavior and content routing. Two heads are considered safely redundant only when their normalized content compositions are close in Aitchison distance and their sink-mass trajectories are also close, avoiding pruning decisions dominated by a shared sink token.
Paper: Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data
arXiv:2608.14712