Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention

What the math gives to ML

The paper models self-attention as a coupled dynamical system in which token states create an interaction graph that immediately changes those same states. Its transferable insight is that a positive within-cluster similarity margin, called the overlap gap, makes cross-cluster attention decay exponentially with inverse temperature and embedding dimension, producing clustered attractors and possible rank or entropy collapse. This suggests an adaptive sharpness controller that estimates token separation and limits attention temperature before the system enters an undesirable condensation regime. The same mechanism can also be used deliberately when stable token grouping is useful for routing or compression.