Towards a mathematical theory of superposition

What the math gives to ML

The paper supplies a concrete superposition mechanism: represent a high-dimensional sparse feature vector through an overcomplete dictionary and decode it with a single ReLU applied to the dictionary Gram operator. The transferable asset is the connection between recoverability and dictionary geometry, especially normalization, near-tightness, and low mutual coherence, rather than sparse coding as a generic objective. A practical neural-network adaptation is a coherence-aware overcomplete bottleneck for MLP or embedding activations, with analytically chosen ReLU biases that suppress cross-feature interference and an explicit coherence regularizer. This is experimentally testable against a standard sparse autoencoder by measuring support precision, reconstruction error, and downstream task accuracy at the same bottleneck width.