# Affine mean-width normalization

- ID: 1604
- Canonical URL: https://synthcore.org/idea/1604/affine-mean-width-normalization
- API JSON: https://synthcore.org/api/idea/1604.json
- API Markdown: https://synthcore.org/api/idea/1604.md
- Verification status: unverified
- Source: [arXiv:2607.29522](https://arxiv.org/abs/2607.29522)
- Category: geometry
- Solves: stability, generalization, memory
- ML areas: embedding, regularization, quantization
- Math tags: convex-analysis, metric-geometry, geometry, linear-algebra
- Ratings: usefulness 5/10; difficulty 6/10; novelty 7/10

## Idea description

Replace or augment LayerNorm on a hidden representation by a learned volume-preserving linear map that minimizes the representation cloud's spherical mean width relative to a volume proxy. The determinant constraint prevents trivial shrinking, so the module targets anisotropy and elongated activation clouds; the expected benefit is a tighter geometry that is easier to quantize or cover with a small codebook.

## Mathematical statement

The paper's main bound states that for every convex body K subset R^n, 1 <= inf_{T in GL(n)} M^*(TK)/vr(TK) <= C sqrt(log(e n)), where GL(n) is the group of invertible n-by-n matrices, M^*(K) is spherical mean width, vr(K)=(vol_n(K)/vol_n(B_2^n))^(1/n) is volume radius, B_2^n is the Euclidean unit ball, and C is universal. Its covering-number theorem states that for every convex body K there is T in GL(n), with vol_n(TK)=vol_n(B_1^n), such that log N(TK,rB_2^n) <= C_cover log N(B_1^n,c_cover rB_2^n) for every r>0; N(K,rB_2^n) is the minimum number of Euclidean radius-r balls covering K, and B_1^n is the crosspolytope. We adapt the determinant constraint det(A)=1 to a finite activation cloud by using K_batch=conv({+x_i,-x_i}), where x_i are centered hidden vectors. The exact convex-hull volume is replaced by a covariance-volume proxy vr_cov(A X)=det(S_A+epsilon I)^(1/(2n)), with S_A the covariance of A x_i. Mean width is estimated from support functions in random spherical directions.

## Key formulas

- $$1\leq\inf_{T\in\mathrm{GL}(n)}\frac{M^{\ast}(TK)}{\operatorname{vr}(TK)}\leq C\sqrt{\log(\mathrm{e}n)},\qquad \operatorname{vr}(K)=\left(\frac{\operatorname{vol}_n(K)}{\operatorname{vol}_n(B_2^n)}\right)^{1/n}.$$
- $$\log N\left(TK,rB_2^n\right)\leq C_{\mathrm{cover}}\log N\left(B_1^n,c_{\mathrm{cover}}rB_2^n\right),\quad \operatorname{vol}_n(TK)=\operatorname{vol}_n(B_1^n).$$
- $$\widehat M^{\ast}(A;X)=\frac{1}{m}\sum_{j=1}^{m}\max_{1\leq i\leq b}\left|u_j^{\mathsf T}A x_i\right|,\quad u_j\sim\operatorname{Unif}(S^{n-1}),\qquad \widehat{\operatorname{vr}}_{\mathrm{cov}}(A;X)=\det\left(S_A+\varepsilon I\right)^{1/(2n)}.$$
- $$\mathcal L_{\mathrm{geom}}(A;X)=\frac{\widehat M^{\ast}(A;X)}{\widehat{\operatorname{vr}}_{\mathrm{cov}}(A;X)}+\lambda\left\|A^{\mathsf T}A-I\right\|_F^2,\qquad A=\exp\left(B-\frac{\operatorname{tr}(B)}{n}I\right).$$

## Implementation notes

Integrate the module immediately after a Transformer MLP or attention output and before residual addition, initially with hidden dimension n=256 or n=512. Maintain an unconstrained trainable matrix B, but construct A=exp(B-tr(B)I/n), which enforces det(A)=1 exactly; for a cheaper implementation, parameterize A using several Householder factors and diagonal log-scales shifted to have zero sum. For each minibatch X={x_i}_{i=1}^b, subtract the batch mean, compute y_i=A x_i, and form S_A=(1/b) sum_i y_i y_i^T. Draw m=16--32 normalized Gaussian vectors u_j, compute h_j=max_i |u_j^T y_i|, and average them to estimate mean width. Compute the covariance volume with a stable Cholesky log-determinant: vr_cov=exp(logdet(S_A+epsilon I)/(2n)). Add lambda times the ratio of these quantities to the task loss, together with a small penalty lambda_id ||A^T A-I||_F^2 to keep optimization near the identity. Replace the hard maximum by tau log(sum_i exp(|u_j^T y_i|/tau)) during warm-up and anneal tau. Update A every 100 optimizer steps if full matrix exponentials are expensive. The paper provides the determinant-preserving affine objective and mean-width/covering rationale; the covariance determinant and random-support estimator are practical proxies rather than exact theorem guarantees. The first experiment should use a 6-layer small Transformer on WikiText-103, comparing the module with LayerNorm, RMSNorm, and whitening at equal parameter count and FLOPs. Track training loss, activation covariance condition number, empirical covering numbers at several radii, and 4-bit or 8-bit activation quantization error. Success is lower activation condition number and quantization error without perplexity degradation, or faster loss descent at equal compute.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.