Log-Finite-Part Polynomial Interaction Kernel

Integrate this into a polynomial-interaction MLP or attention block, specifically at the scalar score computation between two feature vectors rather than in the softmax itself. Let \(x_i,y_j\in\mathbb{R}^d\) be normalized feature vectors, define \(b_{ij}=x_i^\top y_j\), \(q_i=\|x_i\|^2\), and \(r_j=\|y_j\|^2\), and use \(D_{ij}(\rho)=1+\rho b_{ij}+\rho^2q_ir_j\). Choose a small integer \(u\in\{1,2,3\}\) and truncate all powers in \(\rho\) to degree \(K\). Compute \(D_{ij}^u\) by polynomial multiplication, clamp \(D_{ij}\) to \(\delta>0\) before taking the logarithm, and form \(s_{ij}=\operatorname{Coeff}_{\rho^K}[D_{ij}^u(c_u-\tfrac12\log D_{ij})]\), where \(c_u=(H_u-\gamma-\log\pi)/2\). Add \(s_{ij}\) to a standard dot-product score with a learned scalar gate initialized at zero, or use it as the score in a small polynomial-kernel attention head. The paper supplies the continuation and finite-part formula; the implementation estimates no spectral quantities, but should empirically tune \(\rho\), \(K\), \(u\), and the clamp \(\delta\). Use float32 initially and monitor the fraction of clamped entries and score variance. The first experiment should use a 2-layer MLP with a degree-4 polynomial interaction on CIFAR-10 or a tiny transformer on MNIST-adding, comparing dot-product interactions, ridge-regularized polynomial interactions, and the gated logarithmic kernel at equal parameter count. Success means fewer NaNs and less sensitivity to the ridge coefficient, together with lower validation loss or better accuracy at equal FLOPs; a negative result is a rapidly growing clamped-entry fraction or no gain over ordinary ridge regularization.