Uniform Sparsity-Distortion Adapter

Integrate the regularizer into a linear representation adapter, sparse-autoencoder dictionary map, or the first linear layer after a frozen encoder. Let the source feature tensor be \(U\in\mathbb R^{B\times d}\), let \(T\in\mathbb R^{d'\times d}\) be the trainable adapter, and let \(V\) be target features if feature matching is available. First normalize sampled directions \(u_j\) to unit \(\ell_2\) norm. Use minibatch activation vectors, individual examples, feature basis vectors, and random linear combinations of activation vectors so that the constraint is not tied to one observed basis. Compute soft supports using \(s_\tau(z)=\sum_i(1-\exp(-z_i^2/\tau^2))\), then compute ratios \(r_j=(s_\tau(Tu_j)+\epsilon)/(s_\tau(u_j)+\epsilon)\). Add the range loss \(L_{\mathrm{dist}}=\log(\max r_j+\epsilon)-\log(\min r_j+\epsilon)\) to the task or feature-matching loss. Use log-sum-exp soft extrema during early training if hard max/min produces unstable gradients. Initialize \(T\) as identity for equal widths or an orthogonal rectangular map for unequal widths. Anneal \(\tau\) from approximately 0.2 to 0.03 times the median absolute activation. The exact mathematics supplies the max/min distortion criterion; sampled directions and soft support values are empirical approximations. The first experiment should align two 256-dimensional sparse-autoencoder representations on CIFAR-10 or a small Transformer residual stream. Compare MSE-only, MSE plus L1 sparsity, and MSE plus this distortion loss. Measure task accuracy, activation density after a fixed threshold, the variance and max/min range of per-direction sparsity ratios, and pruning accuracy. Success is equal accuracy at lower activation density, improved transfer accuracy at fixed density, or a substantially smaller sparsity-ratio range without increasing training FLOPs materially.