{
 "artifacts": [
  {
   "name": "particle_experiment.py",
   "url": "https://synthcore.org/code/40/particle_experiment.py"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/40/report.md"
  },
  {
   "name": "results.json",
   "url": "https://synthcore.org/code/40/results.json"
  }
 ],
 "category": "dynamics",
 "description": "Replace a wide fixed-feature layer by a finite empirical distribution of trainable feature particles and update the particles with noisy mean-field Langevin dynamics. Use high temperature to explore feature space and anneal toward low temperature so that particles concentrate around predictive directions without immediately collapsing to a single neuron.",
 "download_zip": "https://synthcore.org/download/40",
 "formulas_latex": [
  "$$f_{\\nu}(x)=\\mathbb{E}\\left[\\alpha\\,\\sigma\\bigl(\\langle W,x\\rangle\\bigr)\\right]$$",
  "$$\\nu_M=\\frac{1}{M}\\sum_{i=1}^{M}\\delta_{(a_i,w_i)},\\qquad f_M(x)=\\frac{1}{M}\\sum_{i=1}^{M}a_i\\sigma(w_i^\\top x)$$",
  "$$z_i^{t+1}=\\Pi_{[-A,A]\\times\\mathcal{W}}\\left(z_i^t-\\eta\\,\\widehat{\\nabla}_{z_i}\\left[\\frac{1}{B}\\sum_{b=1}^{B}\\ell\\bigl(f_M(x_b),y_b\\bigr)\\right]+\\sqrt{2\\eta\\lambda_t}\\,\\xi_i^t\\right),\\quad \\xi_i^t\\sim\\mathcal{N}(0,I)$$",
  "$$\\lambda_t=\\lambda_{\\min}+\\frac{1}{2}(\\lambda_{\\max}-\\lambda_{\\min})\\left(1+\\cos\\frac{\\pi t}{T_{\\mathrm{anneal}}}\\right)$$"
 ],
 "id": 275,
 "implementation": "Integrate this at the feature-parameter update of a one-hidden-layer MLP or a transformer MLP expansion layer. Represent the layer with \\(M\\) particles \\((a_i,w_i)\\), where \\(w_i\\in\\mathbb{R}^d\\) is the incoming feature vector and \\(a_i\\) is the outgoing scalar; compute the layer output as \\(h(x)=M^{-1}\\sum_i a_i\\sigma(w_i^\\top x)\\). At every minibatch, run ordinary backpropagation to obtain gradients with respect to every particle, then add independent Gaussian noise with variance \\(2\\eta\\lambda_t\\), clip or project \\(a_i\\) to \\([-A,A]\\), and optionally normalize \\(w_i\\) to the unit sphere. Use a cosine temperature schedule from \\(\\lambda_{\\max}\\in\\{0.5,1,2\\}\\) to \\(\\lambda_{\\min}\\in\\{10^{-3},10^{-2}\\}\\); the mathematically motivated diagnostic is whether behavior changes around \\(\\lambda\\approx1\\). The formulas computed exactly are the empirical mean-field predictor and Langevin step; the gradient is estimated by minibatch backpropagation and no density or Wasserstein metric needs to be explicitly estimated. First test on teacher-student Gaussian single-index and multi-index regression, then CIFAR-10 with a small MLP against a deterministic equally wide MLP and Adam. Track test MSE at fixed samples and FLOPs, particle diversity, cosine alignment between learned \\(w_i\\) and teacher directions, and collapse rate. Success is faster loss descent or lower test error at equal width, with low-temperature particles aligning while retaining several distinct modes in the multi-index task.",
 "math_summary": "The paper represents a mean-field neuron model by a probability measure \\(\\nu\\) over output weights \\(\\alpha\\) and feature parameters \\(W\\), with predictor \\(f_\\nu(x)=\\mathbb{E}_{(\\alpha,W)\\sim\\nu}[\\alpha\\,\\sigma(\\langle W,x\\rangle)]\\). Its dynamics are described as the Wasserstein gradient flow of empirical risk regularized by negative entropy, and the low-temperature stationary law concentrates near hidden indices despite the entropy penalty; the abstract reports a sharp transition around temperature \\(\\lambda\\asymp 1\\). For particles \\(z_i=(a_i,w_i)\\), use the finite-particle empirical measure \\(\\nu_M=M^{-1}\\sum_i\\delta_{z_i}\\), empirical prediction \\(f_M(x)=M^{-1}\\sum_i a_i\\sigma(w_i^\\top x)\\), and the practical Langevin update \\(z_i\\leftarrow z_i-\\eta\\widehat{\\nabla}_{z_i}\\mathcal{L}+\\sqrt{2\\eta\\lambda_t}\\,\\xi_i\\), where \\(\\xi_i\\sim\\mathcal{N}(0,I)\\), \\(\\eta\\) is the step size, and \\(\\lambda_t\\) is the temperature schedule. The noise implements the entropy-driven diffusion; annealing exploits the concentration property.",
 "math_tags": [
  "probability",
  "stochastic-processes",
  "optimal-transport",
  "geometry"
 ],
 "ml_areas": [
  "mlp",
  "optimizer",
  "training-dynamics",
  "initialization"
 ],
 "paper": {
  "arxiv_id": "2606.31429",
  "arxiv_url": "https://arxiv.org/abs/2606.31429",
  "summary_what_math_gives_to_ml": "The paper gives a constructive view of feature learning as a two-level object: a learned marginal distribution over feature parameters forms the base, while coefficient functions on that marginal form the fiber. The transferable asset is that mean-field Langevin dynamics can move the feature marginal toward statistically informative directions, while the induced kernel makes the resulting representation explicit and measurable. A practical adaptation is to train a finite particle approximation with entropy-controlled Langevin noise, then fit or jointly train output coefficients; temperature annealing should allow exploration early and concentration near useful features late. The single-index parity result also suggests eliminating antipodal redundancy through paired or quotient-aware neurons.",
  "title": "The Geometry of Statistical Feature Learning in Mean-Field Langevin Dynamics",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 5,
  "usefulness": 7
 },
 "solves": [
  "accuracy",
  "sample-efficiency",
  "stability"
 ],
 "title": "Entropy-Annealed Feature Particle Layer",
 "url": "https://synthcore.org/idea/275/entropy-annealed-feature-particle-layer",
 "verification": {
  "peer_reviewed": false,
  "stage1_mechanism_check": {
   "worked": false,
   "confidence": 9,
   "verdict": "Built an entropy-annealed particle feature layer with projected output weights, cosine temperature annealing, Gaussian Langevin diffusion, and teacher-direction diagnostics. The mathematical checks passed: predictor error was 5.96e-8 and sampled noise variance was within 1.3% of 2ηλ. However, on the two-index ReLU teacher task the particle model reached test MSE 0.1159 versus 0.000136 for the equally wide Adam baseline; it preserved higher feature diversity but had zero particles strongly aligned with teacher directions, so the claimed practical effect was not demonstrated.",
   "metrics": {
    "baseline": "test MSE 0.0001355; final train MSE 0.0000654; mean teacher alignment 0.8967; aligned fraction 87.5%; weight diversity 1.1638; runtime 1.34s",
    "idea": "test MSE 0.115886; final train MSE 0.09726; mean teacher alignment 0.2828; aligned fraction 0%; weight diversity 1.4009; runtime 1.32s"
   },
   "how_to_run": "python3 particle_experiment.py",
   "files": [
    "particle_experiment.py",
    "results.json"
   ],
   "limitations": "Only a small synthetic two-index ReLU regression task was tested; CIFAR-10, single-index regression, temperature sweeps, alternative step sizes, exact non-Adam Langevin discretization, and larger particle populations were not tested."
  },
  "status": "mechanism_failed",
  "status_label": "Mechanism failed",
  "updated_at": "2026-08-29T23:59:43",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": false,
    "tested": true
   }
  }
 }
}