# Role-Filler Attention

- ID: 2951
- Canonical URL: https://synthcore.org/idea/2951/role-filler-attention
- API JSON: https://synthcore.org/api/idea/2951.json
- API Markdown: https://synthcore.org/api/idea/2951.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.30124](https://arxiv.org/abs/2608.30124)
- Category: architecture
- Solves: accuracy, generalization, sample-efficiency
- ML areas: attention, transformer, world-model
- Math tags: tensor-decomposition, linear-algebra, representation-theory
- Ratings: usefulness 8/10; difficulty 5/10; novelty 6/10

## Idea description

Replace dense attention over structured object tokens with attention over role-filler tensor-product representations. A learned query specifies both a role and a filler, retrieves objects matching that binding, extracts a target role, and rebinds the extracted filler into an output object.

## Mathematical statement

A role vector \(\bm r_j\in\mathbb R^{d_r}\) identifies slot \(j\), and a filler vector \(\bm f_j^i\in\mathbb R^{d_f}\) is the value occupying slot \(j\) in object \(i\). Role vectors are assumed orthonormal, \(\bm r_i^\top\bm r_j=\delta_{ij}\), where \(\delta_{ij}\) is the Kronecker delta. Object \(i\) is the order-2 tensor \(\bm{\mathsfit O}_i=\sum_j\bm r_j\otimes\bm f_j^i\), with \((\bm v\otimes\bm u)_{ab}=v_a u_b\). Contracting an object with a role recovers the corresponding filler: \(\bm r_t^\top\bm{\mathsfit O}_i=\bm f_t^i\). A memory is a superposition \(\bm{\mathsfit M}=\sum_t\bm{\mathsfit O}_t\otimes\bm{\mathsfit O}_t\), where \(t\) indexes stored objects. For a role-filler query \(\bm q=\bm r_m\otimes\bm f_m\), matching can be implemented by the contraction \(s_t=\langle\bm q,\bm{\mathsfit O}_t\rangle\). Soft selection is \(a_t=\operatorname{softmax}_t(s_t/\tau)\), with temperature \(\tau>0\). Extracting target role \(\bm r_n\) and rebinding gives \(\bm{\mathsfit Y}=\sum_t a_t\,\bm r_n\otimes(\bm r_n^\top\bm{\mathsfit O}_t)\).

## Key formulas

- $${\bm{\mathsfit{O}}}_{i}=\sum_{j}{\bm{r}}_{j}\otimes{\bm{f}}_{j}^{i},\qquad {\bm{r}}_{i}^{\top}{\bm{r}}_{j}=\delta_{ij}.$$
- $${\bm{r}}_{t}^{\top}{\bm{\mathsfit{O}}}_{i}=\sum_j({\bm r}_t^\top{\bm r}_j){\bm f}_j^i={\bm f}_t^i.$$
- $${\bm{\mathsfit{M}}}=\sum_t{\bm{\mathsfit{O}}}_{t}\otimes{\bm{\mathsfit{O}}}_{t},\qquad (\bm v\otimes\bm u)_{ij}=v_i u_j.$$
- $$s_t=\left\langle\bm r_m\otimes\bm f_m,\bm{\mathsfit O}_t\right\rangle,\quad a_t=\operatorname{softmax}_t(s_t/\tau),\quad \bm{\mathsfit Y}=\sum_ta_t\,\bm r_n\otimes(\bm r_n^\top\bm{\mathsfit O}_t).$$

## Implementation notes

Integrate this module between a feature encoder and an ordinary Transformer block. Use a batch of structured objects with shape [B,N,R,D], where B is batch size, N is the number of objects, R is the number of roles, and D is filler dimension. Store fixed or learned role vectors R in R^(R by d_r), normalized row-wise; project each object-role feature into fillers F[b,n,r] in R^D. For each attention head, produce a query role distribution and query filler vector from the current token. The simplest MVP uses a discrete target role m, target filler q_f, and output role n. Compute matching scores with scores[b,n] = dot(q_f, F[b,n,m]); this is the orthonormal-role specialization of the paper's tensor contraction. Apply a = softmax(scores / tau, dim=N), extract out_f[b] = sum_n a[b,n] * F[b,n,n_target], and write the result into the output slot as Y[b,n_target] += out_f[b]. For learned soft roles, replace indexing by a full contraction over the role vectors. Estimate nothing from the paper beyond tensor contractions; tune filler normalization and tau empirically. First experiment: a 2-layer MLP encoder plus this module on colored-shape-position objects, training on all but held-out factor combinations. Compare equal-parameter dense self-attention and an MLP using the same optimizer, steps, and FLOPs. Pre-register that held-out-combination accuracy exceeds dense attention by at least 10 percentage points, seen-combination accuracy stays within 3 points, and attention entropy for exact role queries is at least 0.2 nats lower than for mismatched-role queries. Remove only tensor matching by replacing it with a learned dense dot product over flattened object features. Falsification is no held-out gain, no role-selective entropy gap, or degradation larger than 5 points on interpolation tasks.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built a deterministic NumPy MVP for role-filler tensor construction, role contraction, soft retrieval, target-role extraction, and rebinding. Orthonormal contraction error was 1.33e-15; exact queries selected the correct object with 0.995 accuracy across 2,000 episodes, versus 0.579 for the flattened dense control, and rebinding error was 0.035 versus 2.38. Exact-query entropy was consistently lower than mismatched-role entropy, although this is a toy retrieval result rather than evidence for end-to-end neural generalization.

### Mechanism check

- Verdict: Built a deterministic NumPy MVP for role-filler tensor construction, role contraction, soft retrieval, target-role extraction, and rebinding. Orthonormal contraction error was 1.33e-15; exact queries selected the correct object with 0.995 accuracy across 2,000 episodes, versus 0.579 for the flattened dense control, and rebinding error was 0.035 versus 2.38. Exact-query entropy was consistently lower than mismatched-role entropy, although this is a toy retrieval result rather than evidence for end-to-end neural generalization.
- Confidence: 8/10
- Limitations: The experiment is a small NumPy episodic retrieval benchmark, not a trained encoder/Transformer integration. The dense comparison is a simple flattened dot-product control rather than a parameter-matched learned self-attention model; no held-out colored-shape dataset, FLOP matching, GPU timing, or large-scale generalization test was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.9927
- Paired wins: 3/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/1133/bench_report.json)
- [report.md](https://synthcore.org/code/1133/report.md)
- [report_bench_2026-09-02T121342.md](https://synthcore.org/code/1133/report_bench_2026-09-02T121342.md)
- [results.json](https://synthcore.org/code/1133/results.json)
- [role_filler_bench.py](https://synthcore.org/code/1133/role_filler_bench.py)
- [role_filler_experiment.py](https://synthcore.org/code/1133/role_filler_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/1133)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.