Role-Filler Attention
Source paper: TPR-Attention for Combinatorial Generalization arXiv:2608.30124 ⓘ · analyzed Sep 2, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace dense attention over structured object tokens with attention over role-filler tensor-product representations. A learned query specifies both a role and a filler, retrieves objects matching that binding, extracts a target role, and rebinds the extracted filler into an output object.
Formulas
Mathematical statement
A role vector \(\bm r_j\in\mathbb R^{d_r}\) identifies slot \(j\), and a filler vector \(\bm f_j^i\in\mathbb R^{d_f}\) is the value occupying slot \(j\) in object \(i\). Role vectors are assumed orthonormal, \(\bm r_i^\top\bm r_j=\delta_{ij}\), where \(\delta_{ij}\) is the Kronecker delta. Object \(i\) is the order-2 tensor \(\bm{\mathsfit O}_i=\sum_j\bm r_j\otimes\bm f_j^i\), with \((\bm v\otimes\bm u)_{ab}=v_a u_b\). Contracting an object with a role recovers the corresponding filler: \(\bm r_t^\top\bm{\mathsfit O}_i=\bm f_t^i\). A memory is a superposition \(\bm{\mathsfit M}=\sum_t\bm{\mathsfit O}_t\otimes\bm{\mathsfit O}_t\), where \(t\) indexes stored objects. For a role-filler query \(\bm q=\bm r_m\otimes\bm f_m\), matching can be implemented by the contraction \(s_t=\langle\bm q,\bm{\mathsfit O}_t\rangle\). Soft selection is \(a_t=\operatorname{softmax}_t(s_t/\tau)\), with temperature \(\tau>0\). Extracting target role \(\bm r_n\) and rebinding gives \(\bm{\mathsfit Y}=\sum_t a_t\,\bm r_n\otimes(\bm r_n^\top\bm{\mathsfit O}_t)\).
Implementation notes
Integrate this module between a feature encoder and an ordinary Transformer block. Use a batch of structured objects with shape [B,N,R,D], where B is batch size, N is the number of objects, R is the number of roles, and D is filler dimension. Store fixed or learned role vectors R in R^(R by d_r), normalized row-wise; project each object-role feature into fillers F[b,n,r] in R^D. For each attention head, produce a query role distribution and query filler vector from the current token. The simplest MVP uses a discrete target role m, target filler q_f, and output role n. Compute matching scores with scores[b,n] = dot(q_f, F[b,n,m]); this is the orthonormal-role specialization of the paper's tensor contraction. Apply a = softmax(scores / tau, dim=N), extract out_f[b] = sum_n a[b,n] * F[b,n,n_target], and write the result into the output slot as Y[b,n_target] += out_f[b]. For learned soft roles, replace indexing by a full contraction over the role vectors. Estimate nothing from the paper beyond tensor contractions; tune filler normalization and tau empirically. First experiment: a 2-layer MLP encoder plus this module on colored-shape-position objects, training on all but held-out factor combinations. Compare equal-parameter dense self-attention and an MLP using the same optimizer, steps, and FLOPs. Pre-register that held-out-combination accuracy exceeds dense attention by at least 10 percentage points, seen-combination accuracy stays within 3 points, and attention entropy for exact role queries is at least 0.2 nats lower than for mismatched-role queries. Remove only tensor matching by replacing it with a learned dense dot product over flattened object features. Falsification is no held-out gain, no role-selective entropy gap, or degradation larger than 5 points on interpolation tasks.
Verification
Stage 1 · Toy mechanism gate: Passed ✓
Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗
Stage 2 · Practical benchmark result: No measurable effect
Stage 1 — Mechanism check agent confidence 8/10
Built a deterministic NumPy MVP for role-filler tensor construction, role contraction, soft retrieval, target-role extraction, and rebinding. Orthonormal contraction error was 1.33e-15; exact queries selected the correct object with 0.995 accuracy across 2,000 episodes, versus 0.579 for the flattened dense control, and rebinding error was 0.035 versus 2.38. Exact-query entropy was consistently lower than mismatched-role entropy, although this is a toy retrieval result rather than evidence for end-to-end neural generalization.
- Agent confidence
- 8/10
- Baseline
- Flattened dense dot-product control: accuracy 0.579 (repeat 0.551), rebinding L2 error 2.383 (repeat 2.553), entropy 0.069 (repeat 0.073).
- Idea
- Role-filler retrieval: accuracy 0.995 (repeat 0.995), rebinding L2 error 0.0348 (repeat 0.0434), entropy 0.00555 (mismatched-role 0.2085; repeat 0.00983 vs 0.1806). Role contraction maximum absolute error: 1.33e-15.
Limitations: The experiment is a small NumPy episodic retrieval benchmark, not a trained encoder/Transformer integration. The dense comparison is a simple flattened dot-product control rather than a parameter-matched learned self-attention model; no held-out colored-shape dataset, FLOP matching, GPU timing, or large-scale generalization test was performed.
How to run: python3 role_filler_experiment.py
Stage 2 — Benchmark vs. tuned baseline
No significant difference vs. the tuned baseline (+0.0%, p=0.9927, wins 3 / 8 paired seeds).
- Benchmark
- Sequence modeling
- Model
- transformer_tiny
- Paired seeds
- 8
- Baseline mean
- 0.5629
- Idea mean
- 0.5631
- Effect (Δ)
- 0.0002 (+0.0%; negative = idea better)
- Wins
- 3 / 8 paired seeds
- p-value
- 0.9927 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- ±4.5%
- Mechanism
- Not confirmed ✗
- Practical effect
- No effect
- Baseline tuning
- swept over 0 configs
Only the built-in sequence forecasting track was tested; no colored-shape held-out-factor benchmark, FLOP/timing study, vision track, or larger-scale training was run. The MVP uses position-indexed role vectors and soft role-conditioned retrieval rather than a full learned role/filler tensor-product query. The mechanism signature measures trained-model role probability mass rather than entropy.
How to run: python3 role_filler_bench.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Stage 2 — Benchmark latest Failed ✗
Agent confidence: 9/10
Implemented an end-to-end Role-Filler Attention replacement on the structurally matched sequence forecasting track, with identical embedding, positional, feed-forward, and output components for baseline and idea systems. Across 8 paired seeds, the idea had MSE 0.563105 versus baseline 0.562902, paired delta +0.000203 and permutation p=0.9927, so there was no improvement. The trained-model mechanism signature also failed quantitatively: the observed exact-versus-mismatched role-mass gap was only 0.00263 versus the predicted 0.2.
- Baseline
- Best dense-attention sweep config lr=0.001, tau=2.0; full 8-seed MSE mean 0.562902, std 0.061053.
- Idea
- Best role-filler config lr=0.001, tau=1.0; full 8-seed MSE mean 0.563105, std 0.058728; paired delta +0.000203; permutation p=0.9927; 3/8 paired wins.
Benchmark result
No significant difference vs. the tuned baseline (+0.0%, p=0.9927, wins 3 / 8 paired seeds).
- Benchmark
- Sequence modeling
- Model
- transformer_tiny
- Paired seeds
- 8
- Baseline mean
- 0.5629
- Idea mean
- 0.5631
- Effect (Δ)
- 0.0002 (+0.0%; negative = idea better)
- Wins
- 3 / 8 paired seeds
- p-value
- 0.9927 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- ±4.5%
- Mechanism
- Not confirmed ✗
- Practical effect
- No effect
- Baseline tuning
- swept over 0 configs
Only the built-in sequence forecasting track was tested; no colored-shape held-out-factor benchmark, FLOP/timing study, vision track, or larger-scale training was run. The MVP uses position-indexed role vectors and soft role-conditioned retrieval rather than a full learned role/filler tensor-product query. The mechanism signature measures trained-model role probability mass rather than entropy.
How to run: python3 role_filler_bench.py
Artifacts
- bench_report.json 4.3 KB View Raw JSON
- report.md 1.5 KB View
- report_bench_2026-09-02T121342.md 3.6 KB View
- results.json 0.9 KB View Raw JSON
- role_filler_bench.py 5.6 KB View Raw JSON
- role_filler_experiment.py 3.9 KB View Raw JSON