Homotopy-Associative Attention

(1) Integration point: add the module after value aggregation in one or two transformer layers. For token states h_i in R^d, use a binary composition m_2(a,b)=MLP_2([a,b,a elementwise-product b]). (2) Pseudocode: for sampled triples (a,b,c), compute left=m_2(m_2(a,b),c), right=m_2(a,m_2(b,c)); compute u_a=sigma(W_a a), u_b=sigma(W_b b), and u_c=sigma(W_c c); set m3=W_o(u_a elementwise-product u_b elementwise-product u_c); set g=sigmoid(w_g transposed times [b,a elementwise-product c]); replace the selected two-hop output by left+g times m3; add the associator loss shown above. (3) Use a low rank r much smaller than d to reduce cost. The paper-derived ingredients are the ternary-only truncation and the requirement that m3 explain the binary associator. Rank, gating, lambda, and sampling are estimated empirically. Sample adjacent positions, top-attended triples, or length-two graph paths. Clip the ternary residual norm to the median binary-composition norm if training is unstable. (4) First experiment: use a 2-layer 4-head Transformer on SCAN or ListOps, comparing against the same model and a parameter-matched unconstrained gated MLP. Measure validation accuracy, length-generalization accuracy, associator error, gradient variance, and wall-clock overhead. Success is improved compositional or extrapolation accuracy with less than 15 percent training-time overhead and a nonuniform m3 that activates on ambiguous or long-range triples.