# Coherence-aware superposition bottleneck

- ID: 2757
- Canonical URL: https://synthcore.org/idea/2757/coherence-aware-superposition-bottleneck
- API JSON: https://synthcore.org/api/idea/2757.json
- API Markdown: https://synthcore.org/api/idea/2757.md
- Verification status: unverified
- Source: [arXiv:2608.27540](https://arxiv.org/abs/2608.27540)
- Category: architecture
- Solves: accuracy, generalization, stability
- ML areas: mlp, embedding, loss, regularization
- Math tags: linear-algebra, geometry, approximation-theory
- Ratings: usefulness 5/10; difficulty 5/10; novelty 4/10

## Idea description

Insert an overcomplete sparse feature bottleneck into an MLP or embedding stream: encode an activation h with z = ReLU(W^T h + b), then reconstruct or continue computation from Wz. Normalize dictionary columns and train them to remain nearly tight and low-coherence, while choosing a negative bias from an estimate of worst-case cross-feature interference. The hypothesis is that this gives cleaner, more stable feature supports than an ordinary L1 sparse autoencoder at the same latent width.

## Mathematical statement

The paper models a sparse nonnegative feature vector x in R^n using an overcomplete dictionary W = [w_1,...,w_n] in R^(d x n), with h approximately equal to sum_{j in K} x_j w_j = Wx, where K is the active support, |K| <= k, and x_j >= 0. Its decoder is the one-step Gram/ReLU map z = ReLU(W^T W x + b), where b is a bias vector. For normalized columns ||w_i||_2 = 1, define the mutual coherence mu(W) = max_{i != j} |w_i^T w_j|. Since (W^T h)_i = x_i + sum_{j != i} (w_i^T w_j)x_j, the off-diagonal term is bounded in magnitude by mu(W) sum_{j != i} x_j. For a uniform nonnegative coefficient bound 0 <= x_j <= A and support size at most k, a conservative inactive-coordinate suppression bias is b_i = -mu(W) k A. An active coordinate i remains positive whenever its coefficient exceeds the worst-case interference plus the bias magnitude, namely x_i > mu(W)(2k-1)A. The implementation uses the paper's exact ReLU Gram decoder and adapts this coherence bound into a bias and regularization rule. Here h is the network activation, z is the overcomplete feature code, W is trainable, d is the original width, n >> d is the feature width, k is the target maximum support, A is an empirical coefficient cap, and mu(W) is estimated directly from normalized columns.

## Key formulas

- $$h\approx\sum_{j\in K}x_jw_j=Wx,\qquad K\subseteq\{1,\ldots,n\},\quad |K|\le k,\quad x_j\ge 0.$$
- $$z=\operatorname{ReLU}(W^{\top}Wx+b).$$
- $$\mu(W)=\max_{i\ne j}\frac{|w_i^{\top}w_j|}{\|w_i\|_2\|w_j\|_2},\qquad \|w_i\|_2=1\Rightarrow (W^{\top}Wx)_i=x_i+\sum_{j\ne i}(w_i^{\top}w_j)x_j.$$
- $$\mathcal{L}=\|h-Wz\|_2^2+\lambda\|z\|_1+\rho\sum_{i\ne j}(w_i^{\top}w_j)^2,\qquad b_i=-\widehat\mu\,k\,\widehat A.$$

## Implementation notes

1. Integration point: choose an MLP hidden state h in R^d, preferably after LayerNorm and before the next linear layer. Add an overcomplete feature code z in R^n with n = 4d or 8d, compute u = W^T h + b, z = ReLU(u), and either reconstruct h_hat = Wz for an auxiliary sparse autoencoder loss or replace the original hidden state by h_hat during a controlled ablation. Initialize W with Gaussian columns followed by per-column L2 normalization. 2. Pseudocode: normalize columns W; compute the Gram off-diagonal G = W^T W - I; estimate mu_hat = max(abs(G)); maintain A_hat as the 99th percentile of positive entries of z from a calibration batch; set b = -mu_hat * k * A_hat; for each batch, z = relu(W.T @ h + b), h_hat = W @ z, and optimize task_loss + alpha*||h-h_hat||^2 + lambda*mean(abs(z)) + rho*mean(G^2). Recompute mu_hat and A_hat every 100 to 1000 steps with stop-gradient through these statistics, and EMA-smooth b to avoid abrupt changes. 3. Computed from the mathematics: column normalization, the Gram matrix, mutual coherence, the interference-based bias, and the reconstruction decoder. Estimated empirically: A_hat, useful sparsity k, and the tradeoff coefficients alpha, lambda, and rho. 4. First experiment: train a one- or two-layer MLP or a small Transformer on CIFAR-10 or TinyStories, comparing this bottleneck with a standard L1 sparse autoencoder at equal d, n, and FLOPs. Also test synthetic mixtures with known supports. Record reconstruction MSE, average active features, support precision and recall, downstream accuracy or perplexity, and training stability. Success means higher support precision and lower reconstruction error at the same mean activation count, with no downstream degradation; a useful target is a 10% relative reconstruction improvement or substantially fewer false-positive features at fixed sparsity.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.