# Entropy-Annealed Feature Particle Layer

- ID: 275
- Canonical URL: https://synthcore.org/idea/275/entropy-annealed-feature-particle-layer
- API JSON: https://synthcore.org/api/idea/275.json
- API Markdown: https://synthcore.org/api/idea/275.md
- Verification status: mechanism_failed
- Source: [arXiv:2606.31429](https://arxiv.org/abs/2606.31429)
- Category: dynamics
- Solves: accuracy, sample-efficiency, stability
- ML areas: mlp, optimizer, training-dynamics, initialization
- Math tags: probability, stochastic-processes, optimal-transport, geometry
- Ratings: usefulness 7/10; difficulty 5/10; novelty 5/10

## Idea description

Replace a wide fixed-feature layer by a finite empirical distribution of trainable feature particles and update the particles with noisy mean-field Langevin dynamics. Use high temperature to explore feature space and anneal toward low temperature so that particles concentrate around predictive directions without immediately collapsing to a single neuron.

## Mathematical statement

The paper represents a mean-field neuron model by a probability measure \(\nu\) over output weights \(\alpha\) and feature parameters \(W\), with predictor \(f_\nu(x)=\mathbb{E}_{(\alpha,W)\sim\nu}[\alpha\,\sigma(\langle W,x\rangle)]\). Its dynamics are described as the Wasserstein gradient flow of empirical risk regularized by negative entropy, and the low-temperature stationary law concentrates near hidden indices despite the entropy penalty; the abstract reports a sharp transition around temperature \(\lambda\asymp 1\). For particles \(z_i=(a_i,w_i)\), use the finite-particle empirical measure \(\nu_M=M^{-1}\sum_i\delta_{z_i}\), empirical prediction \(f_M(x)=M^{-1}\sum_i a_i\sigma(w_i^\top x)\), and the practical Langevin update \(z_i\leftarrow z_i-\eta\widehat{\nabla}_{z_i}\mathcal{L}+\sqrt{2\eta\lambda_t}\,\xi_i\), where \(\xi_i\sim\mathcal{N}(0,I)\), \(\eta\) is the step size, and \(\lambda_t\) is the temperature schedule. The noise implements the entropy-driven diffusion; annealing exploits the concentration property.

## Key formulas

- $$f_{\nu}(x)=\mathbb{E}\left[\alpha\,\sigma\bigl(\langle W,x\rangle\bigr)\right]$$
- $$\nu_M=\frac{1}{M}\sum_{i=1}^{M}\delta_{(a_i,w_i)},\qquad f_M(x)=\frac{1}{M}\sum_{i=1}^{M}a_i\sigma(w_i^\top x)$$
- $$z_i^{t+1}=\Pi_{[-A,A]\times\mathcal{W}}\left(z_i^t-\eta\,\widehat{\nabla}_{z_i}\left[\frac{1}{B}\sum_{b=1}^{B}\ell\bigl(f_M(x_b),y_b\bigr)\right]+\sqrt{2\eta\lambda_t}\,\xi_i^t\right),\quad \xi_i^t\sim\mathcal{N}(0,I)$$
- $$\lambda_t=\lambda_{\min}+\frac{1}{2}(\lambda_{\max}-\lambda_{\min})\left(1+\cos\frac{\pi t}{T_{\mathrm{anneal}}}\right)$$

## Implementation notes

Integrate this at the feature-parameter update of a one-hidden-layer MLP or a transformer MLP expansion layer. Represent the layer with \(M\) particles \((a_i,w_i)\), where \(w_i\in\mathbb{R}^d\) is the incoming feature vector and \(a_i\) is the outgoing scalar; compute the layer output as \(h(x)=M^{-1}\sum_i a_i\sigma(w_i^\top x)\). At every minibatch, run ordinary backpropagation to obtain gradients with respect to every particle, then add independent Gaussian noise with variance \(2\eta\lambda_t\), clip or project \(a_i\) to \([-A,A]\), and optionally normalize \(w_i\) to the unit sphere. Use a cosine temperature schedule from \(\lambda_{\max}\in\{0.5,1,2\}\) to \(\lambda_{\min}\in\{10^{-3},10^{-2}\}\); the mathematically motivated diagnostic is whether behavior changes around \(\lambda\approx1\). The formulas computed exactly are the empirical mean-field predictor and Langevin step; the gradient is estimated by minibatch backpropagation and no density or Wasserstein metric needs to be explicitly estimated. First test on teacher-student Gaussian single-index and multi-index regression, then CIFAR-10 with a small MLP against a deterministic equally wide MLP and Adam. Track test MSE at fixed samples and FLOPs, particle diversity, cosine alignment between learned \(w_i\) and teacher directions, and collapse rate. Success is faster loss descent or lower test error at equal width, with low-temperature particles aligning while retaining several distinct modes in the multi-index task.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built an entropy-annealed particle feature layer with projected output weights, cosine temperature annealing, Gaussian Langevin diffusion, and teacher-direction diagnostics. The mathematical checks passed: predictor error was 5.96e-8 and sampled noise variance was within 1.3% of 2ηλ. However, on the two-index ReLU teacher task the particle model reached test MSE 0.1159 versus 0.000136 for the equally wide Adam baseline; it preserved higher feature diversity but had zero particles strongly aligned with teacher directions, so the claimed practical effect was not demonstrated.

### Mechanism check

- Verdict: Built an entropy-annealed particle feature layer with projected output weights, cosine temperature annealing, Gaussian Langevin diffusion, and teacher-direction diagnostics. The mathematical checks passed: predictor error was 5.96e-8 and sampled noise variance was within 1.3% of 2ηλ. However, on the two-index ReLU teacher task the particle model reached test MSE 0.1159 versus 0.000136 for the equally wide Adam baseline; it preserved higher feature diversity but had zero particles strongly aligned with teacher directions, so the claimed practical effect was not demonstrated.
- Confidence: 9/10
- Limitations: Only a small synthetic two-index ReLU regression task was tested; CIFAR-10, single-index regression, temperature sweeps, alternative step sizes, exact non-Adam Langevin discretization, and larger particle populations were not tested.

## Artifacts

- [particle_experiment.py](https://synthcore.org/code/40/particle_experiment.py)
- [report.md](https://synthcore.org/code/40/report.md)
- [results.json](https://synthcore.org/code/40/results.json)
- [Download all files as ZIP](https://synthcore.org/download/40)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.