Gaussian Simplex Classification Head

(1) Integration point: replace the final linear classifier \(Wh+b\) with a simplex classifier acting on the penultimate feature tensor \(h\in\mathbb{R}^p\). Use fixed simplex directions for the first experiment and optionally make the directions trainable with a penalty. The trainable scalar temperature \(\tau>0\) controls score scale. (2) Construct directions by setting \(v_i=e_i-\mathbf{1}/m\in\mathbb{R}^m\), normalizing \(w_i=v_i/\|v_i\|_2\), and embedding the resulting \(m\) vectors into \(\mathbb{R}^p\) when \(p\geq m-1\). During training, sample \(K\) independent noises \(\varepsilon_k\sim\mathcal{N}(0,\sigma^2I_p)\), compute \(s_{ik}=\tau w_i^{\mathsf{T}}(h+\varepsilon_k)\), and minimize the mean cross-entropy across \(k\). Pseudocode: for each batch, compute h; for k=1,...,K, draw epsilon; logits[k]=tau*(h+epsilon)@W.T; loss += cross_entropy(logits[k],labels)/K; update network and tau. (3) If learning \(W\), add \(L_{\mathrm{simplex}}=\|WW^{\mathsf{T}}-G_{\mathrm{simp}}\|_F^2+\alpha\|W^{\mathsf{T}}\mathbf{1}\|_2^2\), where \(G_{\mathrm{simp}}\) has diagonal one and off-diagonal \(-1/(m-1)\), and row-normalize \(W\) after each optimizer step. The paper supplies the Gaussian maximum comparison; estimate actual feature norms, score covariance, and noisy accuracy empirically. (4) First experiment: CIFAR-10 with a width-256 MLP or small ResNet, comparing a standard learned head, a fixed simplex head, and a learned head with the penalty at matched compute. Train with \(\sigma\in\{0,0.05,0.1,0.2\}\), then evaluate clean accuracy, Gaussian feature-noise accuracy, common-corruption accuracy, and low-bit activation quantization. The target signal is equal clean accuracy with lower noisy error, ideally a 5-10% relative error reduction at moderate noise and a smoother accuracy-versus-SNR curve.