# Moment-Cone Interaction Regularizer

- ID: 2421
- Canonical URL: https://synthcore.org/idea/2421/moment-cone-interaction-regularizer
- API JSON: https://synthcore.org/api/idea/2421.json
- API Markdown: https://synthcore.org/api/idea/2421.md
- Verification status: unverified
- Source: [arXiv:2608.16836](https://arxiv.org/abs/2608.16836)
- Category: regularization
- Solves: stability, generalization, accuracy
- ML areas: mlp, transformer, regularization, training-dynamics
- Math tags: convex-analysis, linear-algebra, optimization, approximation-theory
- Ratings: usefulness 5/10; difficulty 4/10; novelty 8/10

## Idea description

Use the lifted convex hull as a training-time regularizer for pairs of nonnegative neural features, encouraging their empirical second- and third-order interaction statistics to lie in the paper's moment cone. This constrains correlations, squares, and cubic cross-moments jointly through PSD inequalities instead of merely penalizing large activations.

## Mathematical statement

For a nonnegative feature pair (x1,x2), X11, X12, and X22 denote second moments, while s1 and s2 denote shadow cubic moments. The 2x2 PSD constraints M1 = [[x2,X12],[X12,s1]] >= 0 and M2 = [[x1,X12],[X12,s2]] >= 0 imply x2 s1 >= X12^2 and x1 s2 >= X12^2. The 3x3 localizing constraints M3 = [[X11,s1,x1-s1],[s1,X12,0],[x1-s1,0,1-X12]] >= 0 and M4 = [[X22,s2,x2-s2],[s2,X12,0],[x2-s2,0,1-X12]] >= 0 couple square moments to the product bound. For a minibatch B, estimate x1=E_B[a], x2=E_B[b], X11=E_B[a^2], X12=E_B[ab], X22=E_B[b^2], s1=E_B[a^2b], and s2=E_B[ab^2]. A differentiable violation penalty is phi(M)=sum_i min(lambda_i(M),0)^2, where lambda_i(M) are eigenvalues. The paper supplies the exact moment-cone constraints; the spectral penalty is an ML adaptation.

## Key formulas

- $$M_1=\begin{pmatrix}x_2&X_{12}\\X_{12}&s_1\end{pmatrix}\succeq0\ \Longrightarrow\ x_2s_1\geq X_{12}^{2},\qquad M_2=\begin{pmatrix}x_1&X_{12}\\X_{12}&s_2\end{pmatrix}\succeq0\ \Longrightarrow\ x_1s_2\geq X_{12}^{2}$$
- $$M_3=\begin{pmatrix}X_{11}&s_1&x_1-s_1\\s_1&X_{12}&0\\x_1-s_1&0&1-X_{12}\end{pmatrix}\succeq0,\qquad M_4=\begin{pmatrix}X_{22}&s_2&x_2-s_2\\s_2&X_{12}&0\\x_2-s_2&0&1-X_{12}\end{pmatrix}\succeq0$$
- $$\mathcal{R}_{\mathrm{MC}}=\sum_{k=1}^{4}\sum_j\min\!\left(\lambda_j(M_k),0\right)^2+\rho\,\mathrm{softplus}(X_{12}-1)^2$$
- $$x_1=\mathbb{E}_B[a],\quad x_2=\mathbb{E}_B[b],\quad X_{11}=\mathbb{E}_B[a^2],\quad X_{12}=\mathbb{E}_B[ab],\quad X_{22}=\mathbb{E}_B[b^2],\quad s_1=\mathbb{E}_B[a^2b],\quad s_2=\mathbb{E}_B[ab^2]$$

## Implementation notes

Apply the regularizer to selected pairs of nonnegative channels after softplus or ReLU, preferably to 8-32 sampled channel pairs per layer. For minibatch tensors A and B, compute x1, x2, X11, X12, and X22 by averaging over batch and spatial or sequence positions. Compute s1 = mean(A**2 * B) and s2 = mean(A * B**2). To reduce minibatch noise, maintain detached exponential moving averages of all seven statistics with decay 0.95, but retain current-batch statistics for the gradient path. Construct M1 through M4 exactly as displayed, calculate symmetric eigendecompositions, and add lambda_mc times the sum of squared negative eigenvalues to the task loss. Normalize A and B so X12 is near one; a practical variant learns a positive scale r and applies the constraints to A/r and B/r. The mathematical computation is PSD feasibility testing; EMA smoothing, channel sampling, scale r, and lambda_mc are empirical. Start with a four-layer CIFAR-10 MLP or a small transformer feed-forward block. Compare no regularizer, activation L2, product clipping, and this penalty at equal FLOPs. Track validation accuracy, calibration, activation-tail quantiles, gradient variance, and the fraction of batches with negative eigenvalues. Success is lower activation and gradient outlier rates with equal or better validation performance.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.