Central-Moment Feature Mixer

Integrate the module into a local attention or graph-convolution block, after computing the K neighbor or token feature vectors and before the ordinary value projection. For each query location, let $x_0\in\mathbb{R}^d$ be the center feature and $x_1,\ldots,x_K$ its neighbors. Compute attention logits $a_j=q(x_0)^Tk(x_j)/\sqrt d$, normalized weights $w=\operatorname{softmax}(a)$, weighted mean $\mu=\sum_jw_jx_j$, and centered vectors $z_j=x_j-\mu$. Accumulate $M_2,M_3,M_4$ using the displayed formulas. Avoid full $d^3$ and $d^4$ tensors: use low-rank learned contractions such as $\phi_3(z)=(W_{31}z)\odot(W_{32}z)\odot(W_{33}z)$ and $\phi_4(z)=\prod_{r=1}^4(W_{4r}z)$, with a final projection for each order. Create independent gates $g_n=\operatorname{sigmoid}(\gamma_n)$, initialize them near $0.1$, and return $y=x_0+\sum_{n=2}^4g_nP_n(M_n)$. Pseudocode is: compute logits and weights; compute $\mu$; set $z_j=x_j-\mu$; accumulate each order branch; project branches separately; gate and sum; add the residual. Exact centering and moment accumulation come from the paper; projection rank, gates, and contraction weights are learned. Measure cross-order leakage by feeding synthetic neighborhoods containing only one polynomial order and projecting the output onto all three order branches. First run CIFAR-10 with a four-layer small vision transformer, comparing a standard mixer, a raw-moment mixer using $x_j$ rather than $z_j$, and the central mixer at matched parameters and FLOPs. Add random constant shifts $x_j\mapsto x_j+b$ at evaluation. Success means substantially smaller shift-induced output discrepancy, lower activation and gradient variance, and equal or better clean accuracy; a concrete target is at least 20% lower shift discrepancy with no more than a 0.5 percentage-point clean-accuracy loss.