{
 "artifacts": [
  {
   "name": "bench_report.json",
   "url": "https://synthcore.org/code/1133/bench_report.json"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/1133/report.md"
  },
  {
   "name": "report_bench_2026-09-02T121342.md",
   "url": "https://synthcore.org/code/1133/report_bench_2026-09-02T121342.md"
  },
  {
   "name": "results.json",
   "url": "https://synthcore.org/code/1133/results.json"
  },
  {
   "name": "role_filler_bench.py",
   "url": "https://synthcore.org/code/1133/role_filler_bench.py"
  },
  {
   "name": "role_filler_experiment.py",
   "url": "https://synthcore.org/code/1133/role_filler_experiment.py"
  }
 ],
 "category": "architecture",
 "description": "Replace dense attention over structured object tokens with attention over role-filler tensor-product representations. A learned query specifies both a role and a filler, retrieves objects matching that binding, extracts a target role, and rebinds the extracted filler into an output object.",
 "download_zip": "https://synthcore.org/download/1133",
 "formulas_latex": [
  "$${\\bm{\\mathsfit{O}}}_{i}=\\sum_{j}{\\bm{r}}_{j}\\otimes{\\bm{f}}_{j}^{i},\\qquad {\\bm{r}}_{i}^{\\top}{\\bm{r}}_{j}=\\delta_{ij}.$$",
  "$${\\bm{r}}_{t}^{\\top}{\\bm{\\mathsfit{O}}}_{i}=\\sum_j({\\bm r}_t^\\top{\\bm r}_j){\\bm f}_j^i={\\bm f}_t^i.$$",
  "$${\\bm{\\mathsfit{M}}}=\\sum_t{\\bm{\\mathsfit{O}}}_{t}\\otimes{\\bm{\\mathsfit{O}}}_{t},\\qquad (\\bm v\\otimes\\bm u)_{ij}=v_i u_j.$$",
  "$$s_t=\\left\\langle\\bm r_m\\otimes\\bm f_m,\\bm{\\mathsfit O}_t\\right\\rangle,\\quad a_t=\\operatorname{softmax}_t(s_t/\\tau),\\quad \\bm{\\mathsfit Y}=\\sum_ta_t\\,\\bm r_n\\otimes(\\bm r_n^\\top\\bm{\\mathsfit O}_t).$$"
 ],
 "id": 2951,
 "implementation": "Integrate this module between a feature encoder and an ordinary Transformer block. Use a batch of structured objects with shape [B,N,R,D], where B is batch size, N is the number of objects, R is the number of roles, and D is filler dimension. Store fixed or learned role vectors R in R^(R by d_r), normalized row-wise; project each object-role feature into fillers F[b,n,r] in R^D. For each attention head, produce a query role distribution and query filler vector from the current token. The simplest MVP uses a discrete target role m, target filler q_f, and output role n. Compute matching scores with scores[b,n] = dot(q_f, F[b,n,m]); this is the orthonormal-role specialization of the paper's tensor contraction. Apply a = softmax(scores / tau, dim=N), extract out_f[b] = sum_n a[b,n] * F[b,n,n_target], and write the result into the output slot as Y[b,n_target] += out_f[b]. For learned soft roles, replace indexing by a full contraction over the role vectors. Estimate nothing from the paper beyond tensor contractions; tune filler normalization and tau empirically. First experiment: a 2-layer MLP encoder plus this module on colored-shape-position objects, training on all but held-out factor combinations. Compare equal-parameter dense self-attention and an MLP using the same optimizer, steps, and FLOPs. Pre-register that held-out-combination accuracy exceeds dense attention by at least 10 percentage points, seen-combination accuracy stays within 3 points, and attention entropy for exact role queries is at least 0.2 nats lower than for mismatched-role queries. Remove only tensor matching by replacing it with a learned dense dot product over flattened object features. Falsification is no held-out gain, no role-selective entropy gap, or degradation larger than 5 points on interpolation tasks.",
 "math_summary": "A role vector \\(\\bm r_j\\in\\mathbb R^{d_r}\\) identifies slot \\(j\\), and a filler vector \\(\\bm f_j^i\\in\\mathbb R^{d_f}\\) is the value occupying slot \\(j\\) in object \\(i\\). Role vectors are assumed orthonormal, \\(\\bm r_i^\\top\\bm r_j=\\delta_{ij}\\), where \\(\\delta_{ij}\\) is the Kronecker delta. Object \\(i\\) is the order-2 tensor \\(\\bm{\\mathsfit O}_i=\\sum_j\\bm r_j\\otimes\\bm f_j^i\\), with \\((\\bm v\\otimes\\bm u)_{ab}=v_a u_b\\). Contracting an object with a role recovers the corresponding filler: \\(\\bm r_t^\\top\\bm{\\mathsfit O}_i=\\bm f_t^i\\). A memory is a superposition \\(\\bm{\\mathsfit M}=\\sum_t\\bm{\\mathsfit O}_t\\otimes\\bm{\\mathsfit O}_t\\), where \\(t\\) indexes stored objects. For a role-filler query \\(\\bm q=\\bm r_m\\otimes\\bm f_m\\), matching can be implemented by the contraction \\(s_t=\\langle\\bm q,\\bm{\\mathsfit O}_t\\rangle\\). Soft selection is \\(a_t=\\operatorname{softmax}_t(s_t/\\tau)\\), with temperature \\(\\tau\u003e0\\). Extracting target role \\(\\bm r_n\\) and rebinding gives \\(\\bm{\\mathsfit Y}=\\sum_t a_t\\,\\bm r_n\\otimes(\\bm r_n^\\top\\bm{\\mathsfit O}_t)\\).",
 "math_tags": [
  "tensor-decomposition",
  "linear-algebra",
  "representation-theory"
 ],
 "ml_areas": [
  "attention",
  "transformer",
  "world-model"
 ],
 "paper": {
  "arxiv_id": "2608.30124",
  "arxiv_url": "https://arxiv.org/abs/2608.30124",
  "summary_what_math_gives_to_ml": "The paper's transferable asset is an explicit role-filler algebra: objects are superpositions of tensor products, so querying a role can recover its filler by contraction rather than asking a dense attention layer to infer slot structure from correlations. This creates an attention variant whose matching and transformation operations are compositional by construction and can therefore be tested on held-out combinations of familiar factors. The most practical transfer is a factorized TPR-attention module for small structured token groups, with implicit contractions replacing materialized high-order tensors to control memory and compute.",
  "title": "TPR-Attention for Combinatorial Generalization",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 6,
  "usefulness": 8
 },
 "solves": [
  "accuracy",
  "generalization",
  "sample-efficiency"
 ],
 "title": "Role-Filler Attention",
 "url": "https://synthcore.org/idea/2951/role-filler-attention",
 "verification": {
  "benchmark": {
   "track": "sequence",
   "model": "transformer_tiny",
   "n_seeds": 8,
   "metric_direction": "lower is better",
   "comparison": {
    "baseline_mean": 0.5629024468362331,
    "idea_mean": 0.5631052926182747,
    "delta_mean": 0.00020284578204154968,
    "per_seed_diffs": [
     0.011652529239654541,
     0.0017193257808685303,
     0.04933422803878784,
     0.0018243789672851562,
     -0.052210092544555664,
     0.0223923921585083,
     -0.017337679862976074,
     -0.015752315521240234
    ],
    "idea_wins": 3,
    "n_pairs": 8,
    "p_value": 0.9927,
    "mde": 0.025187222915370296,
    "mde_rel_pct": 4.47452716841682,
    "verdict": "no measurable effect",
    "system_worked": false
   },
   "baseline": {
    "best_cfg": {
     "lr": 0.001,
     "tau": 2
    },
    "sweep": null,
    "full": {
     "mean": 0.5629024468362331,
     "std": 0.0610525769421287,
     "per_seed": [
      0.639130711555481,
      0.47835078835487366,
      0.5805663466453552,
      0.5667509436607361,
      0.6444448828697205,
      0.4818141460418701,
      0.5961759686470032,
      0.5159857869148254
     ],
     "n": 8
    }
   },
   "idea": {
    "mean": 0.5631052926182747,
    "std": 0.05872795158375719,
    "per_seed": [
     0.6507832407951355,
     0.4800701141357422,
     0.6299005746841431,
     0.5685753226280212,
     0.5922347903251648,
     0.5042065382003784,
     0.5788382887840271,
     0.5002334713935852
    ],
    "n": 8
   },
   "mechanism_signature": {
    "confirmed": false,
    "observed_exact_role_mass": 0.03502586856484413,
    "observed_gap": 0.002629663096740842,
    "observed_mismatched_role_mass": 0.03239620546810329,
    "predicted_exact_minus_mismatched_mass": 0.2,
    "prediction": "role-selective exact queries should be more concentrated than mismatched queries"
   },
   "custom_track": null
  },
  "mechanism_confirmed": false,
  "peer_reviewed": false,
  "practical_verdict": "no_effect",
  "stage1_mechanism_check": {
   "worked": true,
   "confidence": 8,
   "verdict": "Built a deterministic NumPy MVP for role-filler tensor construction, role contraction, soft retrieval, target-role extraction, and rebinding. Orthonormal contraction error was 1.33e-15; exact queries selected the correct object with 0.995 accuracy across 2,000 episodes, versus 0.579 for the flattened dense control, and rebinding error was 0.035 versus 2.38. Exact-query entropy was consistently lower than mismatched-role entropy, although this is a toy retrieval result rather than evidence for end-to-end neural generalization.",
   "metrics": {
    "baseline": "Flattened dense dot-product control: accuracy 0.579 (repeat 0.551), rebinding L2 error 2.383 (repeat 2.553), entropy 0.069 (repeat 0.073).",
    "idea": "Role-filler retrieval: accuracy 0.995 (repeat 0.995), rebinding L2 error 0.0348 (repeat 0.0434), entropy 0.00555 (mismatched-role 0.2085; repeat 0.00983 vs 0.1806). Role contraction maximum absolute error: 1.33e-15."
   },
   "how_to_run": "python3 role_filler_experiment.py",
   "files": [
    "role_filler_experiment.py",
    "results.json"
   ],
   "limitations": "The experiment is a small NumPy episodic retrieval benchmark, not a trained encoder/Transformer integration. The dense comparison is a simple flattened dot-product control rather than a parameter-matched learned self-attention model; no held-out colored-shape dataset, FLOP matching, GPU timing, or large-scale generalization test was performed."
  },
  "status": "failed_benchmark",
  "status_label": "Failed on benchmark",
  "updated_at": "2026-09-02T12:30:10",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": false,
    "tested": true
   },
   "practical_benchmark": {
    "beats_baseline": false,
    "tested": true,
    "verdict": "no_effect"
   },
   "toy_mechanism_gate": {
    "confirmed": true,
    "tested": true
   }
  }
 }
}