Entropy-Annealed Feature Particle Layer

Integrate this at the feature-parameter update of a one-hidden-layer MLP or a transformer MLP expansion layer. Represent the layer with \(M\) particles \((a_i,w_i)\), where \(w_i\in\mathbb{R}^d\) is the incoming feature vector and \(a_i\) is the outgoing scalar; compute the layer output as \(h(x)=M^{-1}\sum_i a_i\sigma(w_i^\top x)\). At every minibatch, run ordinary backpropagation to obtain gradients with respect to every particle, then add independent Gaussian noise with variance \(2\eta\lambda_t\), clip or project \(a_i\) to \([-A,A]\), and optionally normalize \(w_i\) to the unit sphere. Use a cosine temperature schedule from \(\lambda_{\max}\in\{0.5,1,2\}\) to \(\lambda_{\min}\in\{10^{-3},10^{-2}\}\); the mathematically motivated diagnostic is whether behavior changes around \(\lambda\approx1\). The formulas computed exactly are the empirical mean-field predictor and Langevin step; the gradient is estimated by minibatch backpropagation and no density or Wasserstein metric needs to be explicitly estimated. First test on teacher-student Gaussian single-index and multi-index regression, then CIFAR-10 with a small MLP against a deterministic equally wide MLP and Adam. Track test MSE at fixed samples and FLOPs, particle diversity, cosine alignment between learned \(w_i\) and teacher directions, and collapse rate. Success is faster loss descent or lower test error at equal width, with low-temperature particles aligning while retaining several distinct modes in the multi-index task.