Expander Sparse Autoencoders: Parameter-Efficient Dictionaries for Mechanistic Interpretability

What the math gives to ML

The paper turns an overcomplete dictionary into a fixed-support sparse operator whose columns touch only d of the m activation coordinates, reducing learned decoder storage from mn to dn without changing latent width or target sparsity k. The transferable asset is not merely sparsity, but expander-style support diversity: small sets of latent features have many distinct output coordinates, reducing collisions and preserving identifiability of sparse codes. A practical neural-network adaptation is an expander-supported tied autoencoder or latent bottleneck, followed by an OMP-style inference path whose dictionary correlations are computed by gather-and-reduce rather than dense matrix multiplication. The main risks are optimization difficulty from a fixed sparse mask and degradation when the required latent sparsity k is too high relative to the expansion of the mask.