# Implicit Higher-Order TPR Memory

- ID: 2952
- Canonical URL: https://synthcore.org/idea/2952/implicit-higher-order-tpr-memory
- API JSON: https://synthcore.org/api/idea/2952.json
- API Markdown: https://synthcore.org/api/idea/2952.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.30124](https://arxiv.org/abs/2608.30124)
- Category: memory
- Solves: memory, scalability, generalization
- ML areas: attention, memory, retrieval, world-model, graph-nn
- Math tags: tensor-decomposition, linear-algebra, algebra
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Support conjunction queries over multiple roles without explicitly storing a huge tensor of repeated objects. Represent the required higher-order memory through query-dependent contractions, enabling compositional retrieval with memory that scales linearly in the number of objects.

## Mathematical statement

For an order-s object \(\bm{\mathsfit O}_t\), the paper constructs a higher-order memory \(\bm{\mathsfit M}=\sum_t\bm{\mathsfit O}_t^{\otimes j}\), where \(\bm{\mathsfit O}_t^{\otimes j}\) is the j-fold outer product and the dimensional condition for contracting an order-m query is \(j\cdot s-m=n\), with m the query order and n the output order. For order-2 objects, \(\bm{\mathsfit O}_t=\sum_i\bm r_i\otimes\bm f_i^t\). A conjunction query containing m role-filler bindings is \(\bm{\mathsfit Q}=\bigotimes_{k=1}^m\bm q_k\), where \(\bm q_k=\bm r_{a_k}\otimes\bm f_{a_k}\). Instead of materializing the explicit higher-order tensor, compute each object's score directly as \(s_t=\prod_{k=1}^m\langle\bm q_k,\bm{\mathsfit O}_t\rangle\). This factorization follows because contraction distributes over outer products. Soft retrieval uses \(a_t=\operatorname{softmax}_t(s_t/\tau)\), after which any target role can be extracted from the selected objects.

## Key formulas

- $${\bm{\mathsfit{M}}}=\sum_t{\bm{\mathsfit{O}}}_{t}^{\otimes j},\qquad j\cdot s-m=n.$$
- $${\bm{\mathsfit{Q}}}=\bigotimes_{k=1}^{m}{\bm q}_k,\qquad {\bm q}_k={\bm r}_{a_k}\otimes{\bm f}_{a_k}.$$
- $${\bm{\mathsfit{O}}}_{t}=\sum_i({\bm r}_i\otimes{\bm f}_i^t),\qquad s_t=\prod_{k=1}^{m}\left\langle{\bm q}_k,{\bm{\mathsfit O}}_t\right\rangle.$$
- $$a_t=\operatorname{softmax}_t(s_t/\tau),\qquad \bm{\mathsfit Y}=\sum_ta_t\,\bm r_b\otimes(\bm r_b^\top\bm{\mathsfit O}_t).$$

## Implementation notes

Integrate this at the memory or retrieval layer of a Transformer, relational graph network, or object-centric world model. Input objects are F[B,N,R,D]; a conjunction query contains m pairs of role indices a_k and filler queries q_f[k]. Do not construct the paper's explicit order-j tensor, whose storage grows exponentially with tensor order. Instead compute one scalar contraction per query factor: c[k,n] = dot(normalize(q_f[k]), normalize(F[:,n,a_k])). Form the higher-order score score[n] = product_k c[k,n]. For numerical stability, use log_score[n] = sum_k log(max(c[k,n], epsilon)) when similarities are nonnegative, or retain sign bits and sum log(abs(c[k,n]) + epsilon) when signed fillers are required. Normalize with a = softmax(score / tau) and extract the requested output role using out = einsum('bn,bnd->bd', a, F[:,:,target_role,:]). For multiple heads, assign separate conjunction queries and concatenate outputs. The paper provides the higher-order contraction factorization; empirical choices are filler normalization, epsilon, temperature, and whether to use products or sums of log similarities. First cheap experiment: synthetic set reasoning with 4 roles, 8 fillers per role, 32 to 256 objects, and two- or three-factor conjunction queries. Compare dense flattened high-order attention, ordinary single-factor attention, and this factorized module at equal retrieval FLOPs while recording peak memory. Pre-register that peak memory grows linearly with object count, three-factor held-out conjunction accuracy exceeds single-factor attention by at least 15 points, and the top-1 minus top-2 score margin increases by at least 20 percent from two to three factors. Ablate only the product factorization by replacing it with an MLP over concatenated factor similarities. Falsification is superlinear memory, no conjunction-specific accuracy improvement, or a non-increasing score margin.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built an implicit higher-order TPR retrieval MVP with factorized conjunction scores, soft retrieval, and target-role extraction. Explicit tensor contraction matched the factorized product to 4.44e-16 maximum error. Across 32–256 objects, factorized retrieval achieved 100% accuracy versus 0.7–16.3% for single-factor attention, while the storage proxy scaled linearly in object count and explicit higher-order storage grew much faster.

### Mechanism check

- Verdict: Built an implicit higher-order TPR retrieval MVP with factorized conjunction scores, soft retrieval, and target-role extraction. Explicit tensor contraction matched the factorized product to 4.44e-16 maximum error. Across 32–256 objects, factorized retrieval achieved 100% accuracy versus 0.7–16.3% for single-factor attention, while the storage proxy scaled linearly in object count and explicit higher-order storage grew much faster.
- Confidence: 8/10
- Limitations: This is a NumPy toy benchmark, not a trained Transformer or world model. Explicit memory was estimated by scalar-count proxy rather than measured by materializing tensors; no MLP ablation, signed-filler stability study, or FLOP-matched neural baseline was tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.34205
- Paired wins: 5/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/1145/bench_report.json)
- [experiment.py](https://synthcore.org/code/1145/experiment.py)
- [report.md](https://synthcore.org/code/1145/report.md)
- [report_bench_2026-09-02T122656.md](https://synthcore.org/code/1145/report_bench_2026-09-02T122656.md)
- [results.json](https://synthcore.org/code/1145/results.json)
- [run_bench.py](https://synthcore.org/code/1145/run_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1145)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.