Proximal Tensor Attention Refinement

Add this as a compact optional attention head for relational sequence modeling, few-shot classification, or multimodal token fusion. Project each token embedding h_t into three d-dimensional factors u_t,v_t,w_t and normalize them. Parameterize the interaction tensor in CP form with learned factors a_r,b_r,c_r in R^d for rank R. For each query or token group, initialize u from the projected query, v from a weighted key average, and w from a value average. Run one to four unrolled refinement sweeps. Compute a_u=sum_r (b_r^T v)(c_r^T w)a_r, then update u <- normalize(beta_1 u-a_u). Compute a_v=sum_r (a_r^T u)(c_r^T w)b_r and update v similarly; compute a_w=sum_r (a_r^T u)(b_r^T v)c_r and update w. Choose the sign of the tensor energy, or negate A, so that minimizing the energy corresponds to maximizing task-relevant compatibility. Begin with beta_i=1 and tune them jointly, or use learned positive scalars through softplus. The mathematical update is exact for each spherical linear-plus-proximal block; parameter learning uses backpropagation through the unrolled sweeps. Compare against standard multi-head attention with matched parameters and FLOPs on a synthetic ternary-relation task, then test patch-token CIFAR-10 classification. Track validation accuracy, seed variance, interaction energy after every sweep, factor norm drift, and latency. Success is higher relational accuracy or lower seed variance at comparable compute.