{
 "artifacts": [
  {
   "name": "bench_report.json",
   "url": "https://synthcore.org/code/1145/bench_report.json"
  },
  {
   "name": "experiment.py",
   "url": "https://synthcore.org/code/1145/experiment.py"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/1145/report.md"
  },
  {
   "name": "report_bench_2026-09-02T122656.md",
   "url": "https://synthcore.org/code/1145/report_bench_2026-09-02T122656.md"
  },
  {
   "name": "results.json",
   "url": "https://synthcore.org/code/1145/results.json"
  },
  {
   "name": "run_bench.py",
   "url": "https://synthcore.org/code/1145/run_bench.py"
  }
 ],
 "category": "memory",
 "description": "Support conjunction queries over multiple roles without explicitly storing a huge tensor of repeated objects. Represent the required higher-order memory through query-dependent contractions, enabling compositional retrieval with memory that scales linearly in the number of objects.",
 "download_zip": "https://synthcore.org/download/1145",
 "formulas_latex": [
  "$${\\bm{\\mathsfit{M}}}=\\sum_t{\\bm{\\mathsfit{O}}}_{t}^{\\otimes j},\\qquad j\\cdot s-m=n.$$",
  "$${\\bm{\\mathsfit{Q}}}=\\bigotimes_{k=1}^{m}{\\bm q}_k,\\qquad {\\bm q}_k={\\bm r}_{a_k}\\otimes{\\bm f}_{a_k}.$$",
  "$${\\bm{\\mathsfit{O}}}_{t}=\\sum_i({\\bm r}_i\\otimes{\\bm f}_i^t),\\qquad s_t=\\prod_{k=1}^{m}\\left\\langle{\\bm q}_k,{\\bm{\\mathsfit O}}_t\\right\\rangle.$$",
  "$$a_t=\\operatorname{softmax}_t(s_t/\\tau),\\qquad \\bm{\\mathsfit Y}=\\sum_ta_t\\,\\bm r_b\\otimes(\\bm r_b^\\top\\bm{\\mathsfit O}_t).$$"
 ],
 "id": 2952,
 "implementation": "Integrate this at the memory or retrieval layer of a Transformer, relational graph network, or object-centric world model. Input objects are F[B,N,R,D]; a conjunction query contains m pairs of role indices a_k and filler queries q_f[k]. Do not construct the paper's explicit order-j tensor, whose storage grows exponentially with tensor order. Instead compute one scalar contraction per query factor: c[k,n] = dot(normalize(q_f[k]), normalize(F[:,n,a_k])). Form the higher-order score score[n] = product_k c[k,n]. For numerical stability, use log_score[n] = sum_k log(max(c[k,n], epsilon)) when similarities are nonnegative, or retain sign bits and sum log(abs(c[k,n]) + epsilon) when signed fillers are required. Normalize with a = softmax(score / tau) and extract the requested output role using out = einsum('bn,bnd-\u003ebd', a, F[:,:,target_role,:]). For multiple heads, assign separate conjunction queries and concatenate outputs. The paper provides the higher-order contraction factorization; empirical choices are filler normalization, epsilon, temperature, and whether to use products or sums of log similarities. First cheap experiment: synthetic set reasoning with 4 roles, 8 fillers per role, 32 to 256 objects, and two- or three-factor conjunction queries. Compare dense flattened high-order attention, ordinary single-factor attention, and this factorized module at equal retrieval FLOPs while recording peak memory. Pre-register that peak memory grows linearly with object count, three-factor held-out conjunction accuracy exceeds single-factor attention by at least 15 points, and the top-1 minus top-2 score margin increases by at least 20 percent from two to three factors. Ablate only the product factorization by replacing it with an MLP over concatenated factor similarities. Falsification is superlinear memory, no conjunction-specific accuracy improvement, or a non-increasing score margin.",
 "math_summary": "For an order-s object \\(\\bm{\\mathsfit O}_t\\), the paper constructs a higher-order memory \\(\\bm{\\mathsfit M}=\\sum_t\\bm{\\mathsfit O}_t^{\\otimes j}\\), where \\(\\bm{\\mathsfit O}_t^{\\otimes j}\\) is the j-fold outer product and the dimensional condition for contracting an order-m query is \\(j\\cdot s-m=n\\), with m the query order and n the output order. For order-2 objects, \\(\\bm{\\mathsfit O}_t=\\sum_i\\bm r_i\\otimes\\bm f_i^t\\). A conjunction query containing m role-filler bindings is \\(\\bm{\\mathsfit Q}=\\bigotimes_{k=1}^m\\bm q_k\\), where \\(\\bm q_k=\\bm r_{a_k}\\otimes\\bm f_{a_k}\\). Instead of materializing the explicit higher-order tensor, compute each object's score directly as \\(s_t=\\prod_{k=1}^m\\langle\\bm q_k,\\bm{\\mathsfit O}_t\\rangle\\). This factorization follows because contraction distributes over outer products. Soft retrieval uses \\(a_t=\\operatorname{softmax}_t(s_t/\\tau)\\), after which any target role can be extracted from the selected objects.",
 "math_tags": [
  "tensor-decomposition",
  "linear-algebra",
  "algebra"
 ],
 "ml_areas": [
  "attention",
  "memory",
  "retrieval",
  "world-model",
  "graph-nn"
 ],
 "paper": {
  "arxiv_id": "2608.30124",
  "arxiv_url": "https://arxiv.org/abs/2608.30124",
  "summary_what_math_gives_to_ml": "The paper's transferable asset is an explicit role-filler algebra: objects are superpositions of tensor products, so querying a role can recover its filler by contraction rather than asking a dense attention layer to infer slot structure from correlations. This creates an attention variant whose matching and transformation operations are compositional by construction and can therefore be tested on held-out combinations of familiar factors. The most practical transfer is a factorized TPR-attention module for small structured token groups, with implicit contractions replacing materialized high-order tensors to control memory and compute.",
  "title": "TPR-Attention for Combinatorial Generalization",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 6,
  "novelty": 7,
  "usefulness": 7
 },
 "solves": [
  "memory",
  "scalability",
  "generalization"
 ],
 "title": "Implicit Higher-Order TPR Memory",
 "url": "https://synthcore.org/idea/2952/implicit-higher-order-tpr-memory",
 "verification": {
  "benchmark": {
   "track": "sequence",
   "model": "shared_transformer_token_attention",
   "n_seeds": 8,
   "metric_direction": "lower is better",
   "comparison": {
    "baseline_mean": 1.0231029689311981,
    "idea_mean": 1.0160572454333305,
    "delta_mean": -0.007045723497867584,
    "per_seed_diffs": [
     -0.01815617084503174,
     0.022841155529022217,
     0.007326722145080566,
     0.012109637260437012,
     -0.012633919715881348,
     -0.02591574192047119,
     -0.0054711103439331055,
     -0.036466360092163086
    ],
    "idea_wins": 5,
    "n_pairs": 8,
    "p_value": 0.34205,
    "mde": 0.016858466245238783,
    "mde_rel_pct": 1.6477780592162945,
    "verdict": "no measurable effect",
    "system_worked": false
   },
   "baseline": {
    "best_cfg": {
     "lr": 0.003,
     "tau": 0.25
    },
    "sweep": [
     {
      "cfg": {
       "lr": 0.001,
       "tau": 0.25
      },
      "mean": 1.0073408037424088
     },
     {
      "cfg": {
       "lr": 0.001,
       "tau": 0.5
      },
      "mean": 1.0072430819272995
     },
     {
      "cfg": {
       "lr": 0.003,
       "tau": 0.25
      },
      "mean": 0.9897160083055496
     },
     {
      "cfg": {
       "lr": 0.003,
       "tau": 0.5
      },
      "mean": 0.9971326589584351
     },
     {
      "cfg": {
       "lr": 0.01,
       "tau": 0.25
      },
      "mean": 1.041932761669159
     },
     {
      "cfg": {
       "lr": 0.01,
       "tau": 0.5
      },
      "mean": 1.0047774612903595
     }
    ],
    "full": {
     "mean": 1.0231029689311981,
     "std": 0.11072729156070826,
     "per_seed": [
      0.9939484000205994,
      0.8335484266281128,
      1.1223986148834229,
      1.0089685916900635,
      0.9049072861671448,
      1.0517245531082153,
      1.2106995582580566,
      1.0586283206939697
     ],
     "n": 8
    }
   },
   "idea": {
    "mean": 1.0160572454333305,
    "std": 0.10683002426665564,
    "per_seed": [
     0.9757922291755676,
     0.856389582157135,
     1.1297253370285034,
     1.0210782289505005,
     0.8922733664512634,
     1.0258088111877441,
     1.2052284479141235,
     1.0221619606018066
    ],
    "n": 8
   },
   "mechanism_signature": {
    "confirmed": true,
    "observed_correlation": 0.9999999403953552,
    "observed_max_abs_error": 0,
    "predicted_correlation": 1,
    "predicted_max_abs_error": 0,
    "prediction": "Two-factor contraction equals the product of trained token-query similarities."
   },
   "custom_track": null
  },
  "mechanism_confirmed": true,
  "peer_reviewed": false,
  "practical_verdict": "no_effect",
  "stage1_mechanism_check": {
   "worked": true,
   "confidence": 8,
   "verdict": "Built an implicit higher-order TPR retrieval MVP with factorized conjunction scores, soft retrieval, and target-role extraction. Explicit tensor contraction matched the factorized product to 4.44e-16 maximum error. Across 32–256 objects, factorized retrieval achieved 100% accuracy versus 0.7–16.3% for single-factor attention, while the storage proxy scaled linearly in object count and explicit higher-order storage grew much faster.",
   "metrics": {
    "baseline": "Single-factor accuracy was 6.3%, 2.0%, and 0.7% for N=32,128,256 at m=2; 16.3%, 5.0%, and 1.3% at m=3.",
    "idea": "Factorized accuracy was 100% for all tested N and m. Mean factorized margins were 0.504, 0.374, 0.332 for m=2 and 0.756, 0.694, 0.655 for m=3. Factor memory proxy was 2048, 8192, 16384 versus explicit m=3 proxy 8388608, 33554432, 67108864."
   },
   "how_to_run": "python3 experiment.py",
   "files": [
    "experiment.py",
    "results.json"
   ],
   "limitations": "This is a NumPy toy benchmark, not a trained Transformer or world model. Explicit memory was estimated by scalar-count proxy rather than measured by materializing tensors; no MLP ablation, signed-filler stability study, or FLOP-matched neural baseline was tested."
  },
  "status": "mech_ok_no_baseline",
  "status_label": "Mechanism confirmed, baseline not beaten",
  "updated_at": "2026-09-02T12:30:10",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": true,
    "tested": true
   },
   "practical_benchmark": {
    "beats_baseline": false,
    "tested": true,
    "verdict": "no_effect"
   },
   "toy_mechanism_gate": {
    "confirmed": true,
    "tested": true
   }
  }
 }
}