{
 "artifacts": [
  {
   "name": "prototype_mixture_experiment.py",
   "url": "https://synthcore.org/code/44/prototype_mixture_experiment.py"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/44/report.md"
  }
 ],
 "category": "architecture",
 "description": "Replace one proposal distribution per draft slot with a small bank of K proposal prototypes that represent distinct target distributions induced by different hidden within-block paths. A lightweight mode predictor selects or mixes prototypes, reducing the averaging penalty when the target has several recurring continuation modes.",
 "download_zip": "https://synthcore.org/download/44",
 "formulas_latex": [
  "$$K=1:\\quad L_1=\\min_q\\mathbb{E}_{Z}\\left[\\mathrm{TV}(p_Z,q)\\right]$$",
  "$$K\u003e1:\\quad L_K=\\min_{q_1,\\ldots,q_K}\\mathbb{E}_{Z}\\left[\\min_{j}\\mathrm{TV}(p_Z,q_j)\\right]$$",
  "$$q_\\theta(x\\mid h)=\\sum_{j=1}^{K}\\pi_\\theta(j\\mid h)q_j(x),\\qquad \\pi_\\theta(j\\mid h)=\\frac{\\exp(a_j(h))}{\\sum_{\\ell=1}^{K}\\exp(a_\\ell(h))}$$",
  "$$\\mathcal{L}_{\\mathrm{mix}}=\\mathbb{E}_{Z}\\left[\\sum_{j=1}^{K}\\pi_\\theta(j\\mid h)\\mathrm{TV}(p_Z,q_j)\\right]+\\lambda H\\left(\\pi_\\theta(\\cdot\\mid h)\\right)$$"
 ],
 "id": 236,
 "implementation": "Modify the drafter's final vocabulary projection at every block position. Instead of one logits vector, output K logits vectors q_j over the vocabulary plus router logits a_j from the shared hidden state h; compute pi_j=softmax(a)_j and emit either the mixture q_theta=sum_j pi_j q_j or K candidate blocks, one per prototype, if target verification can batch candidates efficiently. During training, generate B target rollouts from the same anchor so each rollout provides a path-specific target distribution p_b at the next slot. Optimize the soft loss in the formula, using exact vocabulary total variation when feasible; for large vocabularies, compute it on the union of target and drafter top-k tokens and add one residual probability bucket. Also compute the oracle diagnostic by assigning each rollout to argmin_j TV(p_b,q_j), which reveals the attainable prototype benefit independently of router quality. Initialize q_j using K-means on target logit or probability vectors, then jointly train the shared drafter, prototype heads, and router. Use K=2 and K=4 with shared hidden computation so parameter growth is mostly in the output heads. First compare against a single-head drafter with identical hidden size on a 1B target and 100M drafter using HumanEval, MBPP, and WikiText continuation. Measure per-slot TV, acceptance length, target calls per generated token, and drafter FLOPs. The expected signal is lower rejection and higher accepted tokens per target call, with gains remaining after accounting for the additional K output projections.",
 "math_summary": "At an anchor, Z denotes the hidden within-block realization and p_Z is the target next-token distribution after that realization. The paper compares one shared proposal with K path-dependent prototypes using L_K=min_{q_1,...,q_K} E_Z[min_j TV(p_Z,q_j)]. For K=1 this is L_1=min_q E_Z[TV(p_Z,q)], the information floor. The prototypes q_1,...,q_K are fixed before rollout, while the index j is selected after observing Z in the paper's oracle analysis. In a neural drafter, replace the unavailable oracle with a learned router pi_theta(j|h) from the observable anchor representation h, and train a mixture q_theta(x|h)=sum_j pi_theta(j|h)q_j(x). A usable soft surrogate is L_mix=E_Z[sum_j pi_theta(j|h)TV(p_Z,q_j)] plus an optional router entropy penalty. The reduction L_1-L_K is the mathematically measurable benefit of multimodal proposals; the difference between oracle assignment and learned routing measures router error.",
 "math_tags": [
  "information-theory",
  "probability",
  "optimization",
  "statistics"
 ],
 "ml_areas": [
  "transformer",
  "inference-speedup",
  "moe-routing"
 ],
 "paper": {
  "arxiv_id": "2608.27339",
  "arxiv_url": "https://arxiv.org/abs/2608.27339",
  "summary_what_math_gives_to_ml": "This paper provides a measurable decomposition of block-drafting failure into an irreducible information floor and an avoidable model gap. The key transferable asset is an empirical variational objective over target rollout paths: it quantifies how much proposal disagreement is caused by hidden within-block realizations rather than poor drafter fitting. This can become a training diagnostic that decides whether to improve the drafter or expose more prefix information through shorter blocks or partially sequential drafting. The same total-variation clustering objective also suggests multi-modal proposal heads for representing several recurring target-distribution modes instead of forcing one unconditional proposal to average incompatible branches.",
  "title": "Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 6,
  "novelty": 6,
  "usefulness": 7
 },
 "solves": [
  "accuracy",
  "speedup",
  "scalability"
 ],
 "title": "Prototype-Mixture Block Drafter",
 "url": "https://synthcore.org/idea/236/prototype-mixture-block-drafter",
 "verification": {
  "peer_reviewed": false,
  "stage1_mechanism_check": {
   "worked": true,
   "confidence": 7,
   "verdict": "Built a self-contained prototype-mixture experiment with exact TV-distance math checks, a shared single-proposal baseline, and learned routed prototype heads for K=2 and K=3. The sanity check verified L1=0.45 versus oracle L2=0, while the held-out synthetic experiment reduced TV from 0.4726 to about 0.3431 for routed prototypes and improved the acceptance proxy from 0.5274 to 0.6569. This is a real signal for multimodal targets, although K=3 provided no meaningful benefit beyond K=2 and output-head cost scales with K.",
   "metrics": {
    "baseline": "Shared proposal: held-out TV 0.4726; acceptance proxy 0.5274.",
    "idea": "K=2 routed: TV 0.3431, acceptance proxy 0.6569, reduction 0.1295; oracle K=2 TV 0.3301. K=3 routed: TV 0.3431, acceptance proxy 0.6569. Exact two-mode check: L1 0.45, oracle L2 0.0."
   },
   "how_to_run": "python3 prototype_mixture_experiment.py",
   "files": [
    "prototype_mixture_experiment.py"
   ],
   "limitations": "This is a synthetic categorical-path experiment, not a transformer drafter or real speculative-decoding benchmark. It does not measure actual target calls, accepted block length, wall-clock speed, or FLOPs; it uses a TV-derived acceptance proxy, tests only one random seed, and does not implement top-k residual approximation or K-means initialization."
  },
  "status": "mechanism_works",
  "status_label": "Mechanism works",
  "updated_at": "2026-08-30T00:12:37",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": true,
    "tested": true
   }
  }
 }
}