# Central-Moment Feature Mixer

- ID: 1326
- Canonical URL: https://synthcore.org/idea/1326/central-moment-feature-mixer
- API JSON: https://synthcore.org/api/idea/1326.json
- API Markdown: https://synthcore.org/api/idea/1326.md
- Verification status: unverified
- Source: [arXiv:2607.23629](https://arxiv.org/abs/2607.23629)
- Category: architecture
- Solves: stability, generalization, accuracy
- ML areas: attention, transformer, graph-nn, regularization
- Math tags: geometry, linear-algebra, tensor-decomposition, representation-theory
- Ratings: usefulness 5/10; difficulty 5/10; novelty 6/10

## Idea description

Replace raw polynomial interactions between neighboring feature vectors with central polynomial interactions computed after subtracting the local feature mean. Keep separate second-, third-, and fourth-order channels and apply independent residual gates to them, so a uniform shift of every feature in a neighborhood cannot create artificial cross-order responses. This is a drop-in higher-order mixer for a small transformer or graph neural network.

## Mathematical statement

The paper defines the weighted mean velocity by $\rho=\sum_i f_i$ and $\rho\bm{u}=\sum_i f_i\bm{\xi}_i$, then centers each velocity using $\bm{c}_i=\bm{\xi}_i-\bm{u}$. Its central tensors are $C^{(n)}_{\alpha_1\cdots\alpha_n}=\sum_i f_i c_{i\alpha_1}\cdots c_{i\alpha_n}$ for $n=2,3,4$. Here $f_i$ are nonnegative weights, $\bm{\xi}_i$ are input vectors, $\bm{u}$ is their weighted mean, and $C^{(n)}$ is the weighted centered moment of order $n$. The key property is shift invariance: replacing every $\bm{\xi}_i$ by $\bm{\xi}_i+\bm{b}$ leaves $\bm{c}_i$ and all central moments unchanged. The neural adaptation uses neighborhood features $x_j\in\mathbb{R}^d$, normalized weights $w_j$, mean $\mu=\sum_jw_jx_j$, centered vectors $z_j=x_j-\mu$, and separate order channels $M_n=\sum_jw_j\phi_n(z_j)$. Independent gates prevent the third- and fourth-order branches from being forced to share a raw-coordinate representation.

## Key formulas

- $$\rho=\sum_i f_i,\qquad \rho\bm{u}=\sum_i f_i\bm{\xi}_i,\qquad \bm{c}_i=\bm{\xi}_i-\bm{u}.$$
- $$C^{(n)}_{\alpha_1\cdots\alpha_n}=\sum_i f_i c_{i\alpha_1}\cdots c_{i\alpha_n},\qquad n=2,3,4.$$
- $$\mu=\sum_{j=1}^{K}w_jx_j,\qquad z_j=x_j-\mu,\qquad \sum_{j=1}^{K}w_j=1,\quad w_j\ge 0.$$
- $$M_2=\sum_jw_j(z_j\otimes z_j),\quad M_3=\sum_jw_j(z_j^{\otimes3}),\quad M_4=\sum_jw_j(z_j^{\otimes4}),\qquad y=x_0+\sum_{n=2}^{4}g_nP_n\!\left(\operatorname{vec}(M_n)\right).$$

## Implementation notes

Integrate the module into a local attention or graph-convolution block, after computing the K neighbor or token feature vectors and before the ordinary value projection. For each query location, let $x_0\in\mathbb{R}^d$ be the center feature and $x_1,\ldots,x_K$ its neighbors. Compute attention logits $a_j=q(x_0)^Tk(x_j)/\sqrt d$, normalized weights $w=\operatorname{softmax}(a)$, weighted mean $\mu=\sum_jw_jx_j$, and centered vectors $z_j=x_j-\mu$. Accumulate $M_2,M_3,M_4$ using the displayed formulas. Avoid full $d^3$ and $d^4$ tensors: use low-rank learned contractions such as $\phi_3(z)=(W_{31}z)\odot(W_{32}z)\odot(W_{33}z)$ and $\phi_4(z)=\prod_{r=1}^4(W_{4r}z)$, with a final projection for each order. Create independent gates $g_n=\operatorname{sigmoid}(\gamma_n)$, initialize them near $0.1$, and return $y=x_0+\sum_{n=2}^4g_nP_n(M_n)$. Pseudocode is: compute logits and weights; compute $\mu$; set $z_j=x_j-\mu$; accumulate each order branch; project branches separately; gate and sum; add the residual. Exact centering and moment accumulation come from the paper; projection rank, gates, and contraction weights are learned. Measure cross-order leakage by feeding synthetic neighborhoods containing only one polynomial order and projecting the output onto all three order branches. First run CIFAR-10 with a four-layer small vision transformer, comparing a standard mixer, a raw-moment mixer using $x_j$ rather than $z_j$, and the central mixer at matched parameters and FLOPs. Add random constant shifts $x_j\mapsto x_j+b$ at evaluation. Success means substantially smaller shift-induced output discrepancy, lower activation and gradient variance, and equal or better clean accuracy; a concrete target is at least 20% lower shift discrepancy with no more than a 0.5 percentage-point clean-accuracy loss.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.