Attention ideas

Hierarchical Polynomial Kernel Attention

Construct a sparse attention variant whose positional interaction kernel is piecewise polynomial rather than an arbitrary softmax score. Store key-value content in exact monomial moments within a spatial or learned-coordinate box hierarchy, then evaluate all queries from those moments without materializing the query-by-key matrix.