# Gaussian Simplex Classification Head

- ID: 890
- Canonical URL: https://synthcore.org/idea/890/gaussian-simplex-classification-head
- API JSON: https://synthcore.org/api/idea/890.json
- API Markdown: https://synthcore.org/api/idea/890.md
- Verification status: unverified
- Source: [arXiv:2607.14087](https://arxiv.org/abs/2607.14087)
- Category: architecture
- Solves: accuracy, stability, generalization
- ML areas: mlp, loss, regularization, initialization
- Math tags: probability, geometry, linear-algebra, convex-analysis
- Ratings: usefulness 6/10; difficulty 4/10; novelty 4/10

## Idea description

Replace the unconstrained final classifier with equal-norm regular-simplex class directions and train it under explicit isotropic Gaussian feature noise. At fixed signal energy and equal class priors, the paper's Gaussian-max theorem predicts that this geometry maximizes finite-noise maximum-likelihood decoding probability, making it a concrete candidate for robust classification heads.

## Mathematical statement

Let \(m\) be the number of classes, \(x_1,\ldots,x_m\) be unit signal vectors, and \(G=(\langle x_i,x_j\rangle)_{i,j}\) their Gram matrix. For signal-to-noise ratio \(\lambda>0\), the average correct decoding probability is \(P_{\mathrm{c}}(x_1,\ldots,x_m;\lambda)=e^{-\lambda^2/2}m^{-1}\mathbb{E}\exp\{\lambda\max_i\xi_i\}\), where \(\xi\sim\mathcal{N}(0,G)\). The paper proves that if \(R\) is a correlation matrix satisfying \(R-\mathbf{1}\mathbf{1}^{\mathsf{T}}/m\succeq0\), then \(\max_iX_i\leq_{\mathrm{st}}\max_iZ_i\), where \(X\sim\mathcal{N}(0,R)\) and the \(Z_i\) are independent standard Gaussians. Stochastic domination means \(\Pr(\max_iX_i>t)\leq\Pr(\max_iZ_i>t)\) for every finite \(t\); equality in distribution occurs only when \(R=I_m\). In the neural adaptation, use class directions satisfying \(\|w_i\|_2=1\), \(w_i^{\mathsf{T}}w_j=-1/(m-1)\) for \(i\neq j\), and \(\sum_iw_i=0\). For feature noise \(\varepsilon\sim\mathcal{N}(0,\sigma^2I)\), logits are \(s_i=\tau w_i^{\mathsf{T}}(h+\varepsilon)\), with effective feature SNR \(\lambda=\tau\|h\|_2/\sigma\). The theorem applies exactly only to the Gaussian score model, so robustness on learned non-Gaussian features is an empirical hypothesis.

## Key formulas

- $P_{\mathrm{c}}(x_{1},\ldots,x_{m};\lambda)=\frac{e^{-\lambda^{2}/2}}{m}\mathbb{E}\exp\!\left\{\lambda\max_{1\leq i\leq m}\xi_{i}\right\},\qquad \xi\sim\mathcal{N}(0,G),\quad G_{ij}=\langle x_i,x_j\rangle.$
- $R-\frac{\mathbf{1}\mathbf{1}^{\mathsf{T}}}{m}\succeq0\quad\Longrightarrow\quad \max_iX_i\leq_{\mathrm{st}}\max_iZ_i,\qquad X\sim\mathcal{N}(0,R),\quad Z_i\stackrel{\mathrm{iid}}{\sim}\mathcal{N}(0,1).$
- $\|w_i\|_2=1,\qquad w_i^{\mathsf{T}}w_j=-\frac{1}{m-1}\ \ (i\neq j),\qquad \sum_{i=1}^{m}w_i=0.$
- $s_i=\tau w_i^{\mathsf{T}}(h+\varepsilon),\qquad \varepsilon\sim\mathcal{N}(0,\sigma^2I_p),\qquad \lambda=\frac{\tau\|h\|_2}{\sigma}.$

## Implementation notes

(1) Integration point: replace the final linear classifier \(Wh+b\) with a simplex classifier acting on the penultimate feature tensor \(h\in\mathbb{R}^p\). Use fixed simplex directions for the first experiment and optionally make the directions trainable with a penalty. The trainable scalar temperature \(\tau>0\) controls score scale. (2) Construct directions by setting \(v_i=e_i-\mathbf{1}/m\in\mathbb{R}^m\), normalizing \(w_i=v_i/\|v_i\|_2\), and embedding the resulting \(m\) vectors into \(\mathbb{R}^p\) when \(p\geq m-1\). During training, sample \(K\) independent noises \(\varepsilon_k\sim\mathcal{N}(0,\sigma^2I_p)\), compute \(s_{ik}=\tau w_i^{\mathsf{T}}(h+\varepsilon_k)\), and minimize the mean cross-entropy across \(k\). Pseudocode: for each batch, compute h; for k=1,...,K, draw epsilon; logits[k]=tau*(h+epsilon)@W.T; loss += cross_entropy(logits[k],labels)/K; update network and tau. (3) If learning \(W\), add \(L_{\mathrm{simplex}}=\|WW^{\mathsf{T}}-G_{\mathrm{simp}}\|_F^2+\alpha\|W^{\mathsf{T}}\mathbf{1}\|_2^2\), where \(G_{\mathrm{simp}}\) has diagonal one and off-diagonal \(-1/(m-1)\), and row-normalize \(W\) after each optimizer step. The paper supplies the Gaussian maximum comparison; estimate actual feature norms, score covariance, and noisy accuracy empirically. (4) First experiment: CIFAR-10 with a width-256 MLP or small ResNet, comparing a standard learned head, a fixed simplex head, and a learned head with the penalty at matched compute. Train with \(\sigma\in\{0,0.05,0.1,0.2\}\), then evaluate clean accuracy, Gaussian feature-noise accuracy, common-corruption accuracy, and low-bit activation quantization. The target signal is equal clean accuracy with lower noisy error, ideally a 5-10% relative error reduction at moderate noise and a smoother accuracy-versus-SNR curve.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.