{
 "artifacts": [
  {
   "name": "experiment.py",
   "url": "https://synthcore.org/code/28/experiment.py"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/28/report.md"
  }
 ],
 "category": "architecture",
 "description": "Represent an attention-head population by trainable particles sampled from a parameter distribution instead of treating heads as unrelated deterministic modules. Train the particles with a Wasserstein gradient-flow discretization, adding isotropic diffusion to maintain head diversity and avoid collapse to redundant heads. At inference, average the particle outputs, with optional particle subsampling or pruning.",
 "download_zip": "https://synthcore.org/download/28",
 "formulas_latex": [
  "$$\\partial_s\\rho=\\nabla_{\\theta}\\cdot\\left(\\rho\\,\\nabla_{\\theta}\\frac{\\delta\\mathcal{L}_{\\lambda}}{\\delta\\rho}\\right)+\\beta\\Delta_{\\theta}\\rho$$",
  "$$d\\theta_i(s)=-\\nabla_{\\theta_i}\\widehat{\\mathcal{L}}_{\\lambda}(\\theta_1,\\ldots,\\theta_M)\\,ds+\\sqrt{2\\beta}\\,dW_i(s)$$",
  "$$V_{\\rho,\\mu}(x)=\\int v_{\\theta}(x;\\mu)\\,\\rho(d\\theta)\\ \\approx\\ V_M(x)=\\frac{1}{M}\\sum_{i=1}^{M}v_{\\theta_i}(x;\\mu)$$",
  "$$\\partial_t\\mu+\\nabla_x\\cdot\\big(\\mu V_{\\rho_s,\\mu}\\big)=0,\\qquad x^{\\ell+1}=x^{\\ell}+\\Delta t\\,\\frac{1}{M}\\sum_{i=1}^{M}v_{\\theta_i}(x^{\\ell};\\mu)$$"
 ],
 "id": 153,
 "implementation": "(1) Integration point: replace a standard multi-head attention block's fixed list of H head modules by M trainable particles \\(\\theta_1,\\ldots,\\theta_M\\). Each particle contains query, key, value, and output projections. For input \\(X\\in\\mathbb{R}^{B\\times N\\times d}\\), compute each head output \\(v_{\\theta_i}(X)\\), average the outputs as \\(Y=M^{-1}\\sum_i v_{\\theta_i}(X)\\), then apply the residual connection and normalization. Represent \\(\\mu\\) by the minibatch token tensor; no density estimation is required. (2) Pseudocode: initialize \\(\\theta_i\\sim\\rho_0\\). For each minibatch, compute \\(Y=M^{-1}\\sum_i v_{\\theta_i}(X)\\), evaluate the task loss \\(L\\), backpropagate to obtain \\(g_i=\\nabla_{\\theta_i}L\\), sample \\(\\epsilon_i\\sim\\mathcal{N}(0,I)\\), and update \\(\\theta_i\\leftarrow\\theta_i-\\eta g_i+\\sqrt{2\\beta\\eta}\\epsilon_i\\). This is Euler-Maruyama for the displayed SDE. Adam may be used for the deterministic term, but record its effective step size before applying the noise. (3) Quantities computed from the mathematics are the particle approximation of \\(\\rho_s\\), the Monte Carlo estimate \\(V_M\\), and the noise amplitude \\(\\sqrt{2\\beta\\eta}\\). Automatic differentiation supplies the finite-particle gradient; the functional derivative is not computed explicitly. Estimate diversity from pairwise cosine similarity of attention maps, particle-output covariance, and effective rank. (4) First experiment: train a 6-layer, 256-dimensional Transformer on WikiText-2, comparing ordinary 8-head attention with M=8, 16, and 32 particles at matched FLOPs or parameter count. Sweep \\(\\beta\\in\\{0,10^{-5},10^{-4},10^{-3}\\}\\), optionally annealing \\(\\beta\\) to zero. Measure validation perplexity, training-loss smoothness, head similarity, gradient variance, and accuracy after dropping half the particles at inference. Success means lower validation perplexity or improved corruption robustness at equal compute, reduced head redundancy, or comparable perplexity with particle subsampling. Excessive diffusion should increase final loss, giving a falsifiable stability-versus-exploration curve.",
 "math_summary": "The paper's training equation is $$\\partial_s\\rho=\\nabla_{\\theta}\\cdot\\left(\\rho\\nabla_{\\theta}\\frac{\\delta\\mathcal{L}_{\\lambda}}{\\delta\\rho}\\right)+\\beta\\Delta_{\\theta}\\rho$$, where \\(s\\) is training time, \\(\\rho_s\\in\\mathcal{P}_2(\\mathbb{R}^p)\\) is the distribution of attention-head parameters \\(\\theta\\in\\mathbb{R}^p\\), \\(\\mathcal{L}_{\\lambda}[\\rho]\\) is the regularized risk, \\(\\delta\\mathcal{L}_{\\lambda}/\\delta\\rho\\) is its first variation, and \\(\\beta\\geq 0\\) is the diffusion or entropy-regularization coefficient. The divergence term is Wasserstein gradient descent on the parameter distribution, while \\(\\beta\\Delta_\\theta\\rho\\) spreads probability mass. A particle approximation is the interacting SDE \\(d\\theta_i=-\\nabla_{\\theta_i}\\widehat{\\mathcal{L}}_{\\lambda}(\\theta_1,\\ldots,\\theta_M)ds+\\sqrt{2\\beta}\\,dW_i(s)\\), where \\(M\\) is the number of particles and \\(W_i\\) are independent Brownian motions. The token mean-field operator is \\(V_{\\rho,\\mu}(x)=\\mathbb{E}_{\\theta\\sim\\rho}[v_\\theta(x;\\mu)]\\), with token state \\(x\\), token distribution \\(\\mu\\), and single-head update \\(v_\\theta\\). Its Monte Carlo approximation is \\(V_M(x)=M^{-1}\\sum_{i=1}^M v_{\\theta_i}(x;\\mu)\\). The depth evolution \\(\\partial_t\\mu+\\nabla_x\\cdot(\\mu V_{\\rho_s,\\mu})=0\\) motivates residual token updates with step size \\(\\Delta t\\).",
 "math_tags": [
  "probability",
  "optimal-transport",
  "stochastic-processes",
  "dynamical-systems"
 ],
 "ml_areas": [
  "attention",
  "transformer",
  "optimizer",
  "regularization"
 ],
 "paper": {
  "arxiv_id": "2608.25055",
  "arxiv_url": "https://arxiv.org/abs/2608.25055",
  "summary_what_math_gives_to_ml": "The paper formulates transformer computation and training as two coupled mean-field evolutions: a token law transported through depth and a probability law over attention-head parameters evolved by Wasserstein gradient flow. The transferable asset is replacing a fixed finite collection of heads with a trainable distribution whose particle approximation naturally supports head diversity, variable-width inference, and principled entropy-induced exploration. A practical first test is a particle attention layer in which each head parameter is updated by the empirical risk gradient plus calibrated Langevin noise, while the layer output averages the particle heads. The available extraction does not expose the paper's detailed attention kernel or convergence constants, so this proposal is an MVP of the stated mean-field mechanism rather than a complete reproduction of every theorem.",
  "title": "A Mean-Field Theory of Transformers: Well-Posedness of the Coupled Data--Parameter Dynamics and Global Convergence of Training",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 5,
  "usefulness": 6
 },
 "solves": [
  "accuracy",
  "generalization",
  "stability"
 ],
 "title": "Langevin Mean-Field Attention Heads",
 "url": "https://synthcore.org/idea/153/langevin-mean-field-attention-heads",
 "verification": {
  "peer_reviewed": false,
  "stage1_mechanism_check": {
   "worked": false,
   "confidence": 9,
   "verdict": "Built an explicit particle-attention module with Euler–Maruyama parameter noise, residual attention updates, diversity diagnostics, particle dropping, and a reproducible synthetic classification benchmark. The diffusion variance matched 2βηT within 0.98% and the deterministic quadratic update matched its expected contraction. However, β=0 achieved 0.3346 validation loss and 95.31% accuracy, while β=0.001 and β=0.005 achieved losses 0.3644 and 0.4203 with accuracies 94.14% and 94.53%; diffusion also reduced half-particle accuracy, so no promised performance win was observed.",
   "metrics": {
    "baseline": "8 deterministic heads (β=0): validation loss 0.3346, accuracy 95.31%, half-head accuracy 94.14%, attention-map cosine 0.4793",
    "idea": "8 Langevin particles: β=0.001 loss 0.3644, accuracy 94.14%, half-particle accuracy 92.58%; β=0.005 loss 0.4203, accuracy 94.53%, half-particle accuracy 90.63%, attention-map cosine 0.3243"
   },
   "how_to_run": "python3 experiment.py",
   "files": [
    "experiment.py"
   ],
   "limitations": "Only a tiny synthetic classification task and 8 particles were tested; no WikiText-2, language modeling, large Transformer, matched-FLOP scaling study, annealing schedule, Adam effective-step analysis, or broader seed sweep was performed."
  },
  "status": "mechanism_failed",
  "status_label": "Mechanism failed",
  "updated_at": "2026-08-29T22:50:59",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": false,
    "tested": true
   }
  }
 }
}