arXiv:2608.30124 2026 Architecture 2 ideas extracted · analyzed Sep 2, 2026
The paper's transferable asset is an explicit role-filler algebra: objects are superpositions of tensor products, so querying a role can recover its filler by contraction rather than asking a dense attention layer to infer slot structure from correlations. This creates an attention variant whose matching and transformation operations are compositional by construction and can therefore be tested on held-out combinations of familiar factors. The most practical transfer is a factorized TPR-attention module for small structured token groups, with implicit contractions replacing materialized high-order tensors to control memory and compute.
Replace dense attention over structured object tokens with attention over role-filler tensor-product representations. A learned query specifies both a role and a filler, retrieves objects matching that binding, extracts a target role, and rebinds the extracted filler into an output object.
Support conjunction queries over multiple roles without explicitly storing a huge tensor of repeated objects. Represent the required higher-order memory through query-dependent contractions, enabling compositional retrieval with memory that scales linearly in the number of objects.