# Stable Magnitude Bottleneck

- ID: 488
- Canonical URL: https://synthcore.org/idea/488/stable-magnitude-bottleneck
- API JSON: https://synthcore.org/api/idea/488.json
- API Markdown: https://synthcore.org/api/idea/488.md
- Verification status: unverified
- Source: [arXiv:2607.06693](https://arxiv.org/abs/2607.06693)
- Category: architecture
- Solves: stability, accuracy, sample-efficiency
- ML areas: vae, embedding, architecture, loss
- Math tags: probability, functional-analysis, statistics
- Ratings: usefulness 5/10; difficulty 4/10; novelty 7/10

## Idea description

Insert a magnitude-only bottleneck whose output is the absolute value of a random independent-feature expansion of the latent vector. Train a decoder to reconstruct the latent representation or input modulo one global sign, while explicitly rejecting feature distributions whose normalized L1 mass is too small. The module provides a controlled way to obtain sign-invariant representations without allowing arbitrary coordinate-wise sign loss.

## Mathematical statement

Definition 1.1 defines stable phase retrieval for a real normed function space X and subspace E: there must exist C < infinity such that min over sigma in {+1,-1} of ||f - sigma g||_X <= C || |f| - |g| ||_X for every f,g in E. Here f and g are functions, |f| is the pointwise magnitude, and C is the noise-amplification constant. The paper's main characterization applies when E is the L2-span of independent centered random variables xi_i, normalized by ||xi_i||_2 = 1: stable phase retrieval holds exactly when all but possibly one coordinate satisfy a uniform two-sided L1 bound, meaning there are constants 0 < a <= b < infinity with a <= ||xi_i||_1 <= b for those coordinates; the upper bound is automatically at most 1 under L2 normalization. We use the lower bound a as an anti-degeneracy condition: if a feature is concentrated near zero, the absolute-value map can erase too much information. For a finite neural bottleneck, z in R^d is encoded as f_z(t) = sum_i z_i xi_i(t), sampled at t_1,...,t_m, and the observed code is y_j = |f_z(t_j)|. The practical surrogate for the theorem's stability inequality is min over sigma in {+1,-1} ||z - sigma z_hat||_2 <= C_hat ||y - y_hat||_2, where z_hat is the decoder's estimate and C_hat is measured on held-out perturbations.

## Key formulas

- $$\min_{\sigma\in\{\pm 1\}}\lVert f-\sigma g\rVert_X\leq C\,\lVert\lvert f\rvert-\lvert g\rvert\rVert_X\qquad\text{for all }f,g\in E.$$
- $$\lVert\xi_i\rVert_2=1,\qquad a\leq\lVert\xi_i\rVert_1\leq b\quad\text{for all but possibly one }i.$$
- $$f_z(t)=\sum_{i=1}^{d}z_i\xi_i(t),\qquad y_j=\lvert f_z(t_j)\rvert=\left\lvert\sum_{i=1}^{d}z_i\xi_i(t_j)\right\rvert,$$
- $$\min_{\sigma\in\{\pm1\}}\lVert z-\sigma\widehat z\rVert_2\leq \widehat C\,\lVert y-\widehat y\rVert_2.$$

## Implementation notes

(1) Integration point: replace the usual linear latent projection in a small VAE, autoencoder, or contrastive encoder with a magnitude bottleneck. Given latent vector z in R^d, maintain m fixed random feature vectors r_j in R^d, where r_{j,i} is sampled independently from a centered distribution such as Rademacher, Gaussian, or a bounded symmetric distribution. Compute u = Rz, y = abs(u), and feed y to the decoder. Use a decoder that outputs z_hat or reconstructs x. The magnitude operation is the only nonlinearity in this module; do not apply independent learned sign flips after it, because the target ambiguity is one global sign.

(2) Pseudocode:
```text
sample or initialize R[j,i] independently and centered
normalize each column so mean_j R[j,i]^2 = 1
for each minibatch x:
    z = encoder(x)                         # R^d
    u = z @ R.T                            # R^m
    y = abs(u)                             # magnitude bottleneck
    z_hat = decoder_latent(y)
    x_hat = decoder(z_hat)
    sigma = sign(sum_i z[i] * z_hat[i])    # stop gradient through sigma
    L_phase = mean(||z - sigma*z_hat||_2^2)
    L_recon = reconstruction_loss(x_hat, x)
    L = L_recon + lambda_phase*L_phase + beta*KL_or_prior_loss
    update network parameters
```

(3) The paper supplies the structural condition and the desired inequality, not a finite-sample value of C. Estimate each feature's empirical normalized L1 mass by a_i = mean_j abs(R[j,i]) / sqrt(mean_j R[j,i]^2), and reject or resample columns with a_i below a threshold such as 0.5. Estimate C_hat by generating pairs z and z + delta, decoding them, and taking the maximum observed ratio min_sigma ||z-sigma*z_hat||_2 / (||y-y_hat||_2+epsilon). Compare this against an unconstrained abs bottleneck using sparse, heavy-tailed, and near-zero feature distributions.

(4) First experiment: use MNIST or CIFAR-10 with a 16-dimensional latent autoencoder and m in {32, 64, 128}; compare a standard linear latent bottleneck, a magnitude bottleneck with Gaussian R, and the proposed L1-screened R. Add Gaussian noise to y at several signal-to-noise ratios and evaluate reconstruction error, latent recovery modulo global sign, and the empirical stability ratio C_hat. The expected signal is lower reconstruction degradation under magnitude noise, fewer catastrophic latent failures, and a monotone relationship between the minimum empirical a_i and robustness. A useful negative result is also falsifiable: if screened features do not improve C_hat or reconstruction at equal m, discard the method.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.