Unverified 2026

Moment-Cone Interaction Regularizer

Implementation & benchmark of arXiv:2608.16836 — Nonnegative Quadratics over a Quadrant with a Bilinear Constraint

Usefulness5/10

Difficulty4/10

Novelty8/10

Regularization Stability Generalization Accuracy Mlp Transformer Regularization Training dynamics Convex analysis Linear algebra Optimization Approximation theory

Source paper: Nonnegative Quadratics over a Quadrant with a Bilinear Constraint arXiv:2608.16836 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use the lifted convex hull as a training-time regularizer for pairs of nonnegative neural features, encouraging their empirical second- and third-order interaction statistics to lie in the paper's moment cone. This constrains correlations, squares, and cubic cross-moments jointly through PSD inequalities instead of merely penalizing large activations.

Formulas

$$M_1=\begin{pmatrix}x_2&X_{12}\\X_{12}&s_1\end{pmatrix}\succeq0\ \Longrightarrow\ x_2s_1\geq X_{12}^{2},\qquad M_2=\begin{pmatrix}x_1&X_{12}\\X_{12}&s_2\end{pmatrix}\succeq0\ \Longrightarrow\ x_1s_2\geq X_{12}^{2}$$

$$M_3=\begin{pmatrix}X_{11}&s_1&x_1-s_1\\s_1&X_{12}&0\\x_1-s_1&0&1-X_{12}\end{pmatrix}\succeq0,\qquad M_4=\begin{pmatrix}X_{22}&s_2&x_2-s_2\\s_2&X_{12}&0\\x_2-s_2&0&1-X_{12}\end{pmatrix}\succeq0$$

$$\mathcal{R}_{\mathrm{MC}}=\sum_{k=1}^{4}\sum_j\min\!\left(\lambda_j(M_k),0\right)^2+\rho\,\mathrm{softplus}(X_{12}-1)^2$$

$$x_1=\mathbb{E}_B[a],\quad x_2=\mathbb{E}_B[b],\quad X_{11}=\mathbb{E}_B[a^2],\quad X_{12}=\mathbb{E}_B[ab],\quad X_{22}=\mathbb{E}_B[b^2],\quad s_1=\mathbb{E}_B[a^2b],\quad s_2=\mathbb{E}_B[ab^2]$$

Mathematical statement

For a nonnegative feature pair (x1,x2), X11, X12, and X22 denote second moments, while s1 and s2 denote shadow cubic moments. The 2x2 PSD constraints M1 = [[x2,X12],[X12,s1]] >= 0 and M2 = [[x1,X12],[X12,s2]] >= 0 imply x2 s1 >= X12^2 and x1 s2 >= X12^2. The 3x3 localizing constraints M3 = [[X11,s1,x1-s1],[s1,X12,0],[x1-s1,0,1-X12]] >= 0 and M4 = [[X22,s2,x2-s2],[s2,X12,0],[x2-s2,0,1-X12]] >= 0 couple square moments to the product bound. For a minibatch B, estimate x1=E_B[a], x2=E_B[b], X11=E_B[a^2], X12=E_B[ab], X22=E_B[b^2], s1=E_B[a^2b], and s2=E_B[ab^2]. A differentiable violation penalty is phi(M)=sum_i min(lambda_i(M),0)^2, where lambda_i(M) are eigenvalues. The paper supplies the exact moment-cone constraints; the spectral penalty is an ML adaptation.

Implementation notes

Apply the regularizer to selected pairs of nonnegative channels after softplus or ReLU, preferably to 8-32 sampled channel pairs per layer. For minibatch tensors A and B, compute x1, x2, X11, X12, and X22 by averaging over batch and spatial or sequence positions. Compute s1 = mean(A**2 * B) and s2 = mean(A * B**2). To reduce minibatch noise, maintain detached exponential moving averages of all seven statistics with decay 0.95, but retain current-batch statistics for the gradient path. Construct M1 through M4 exactly as displayed, calculate symmetric eigendecompositions, and add lambda_mc times the sum of squared negative eigenvalues to the task loss. Normalize A and B so X12 is near one; a practical variant learns a positive scale r and applies the constraints to A/r and B/r. The mathematical computation is PSD feasibility testing; EMA smoothing, channel sampling, scale r, and lambda_mc are empirical. Start with a four-layer CIFAR-10 MLP or a small transformer feed-forward block. Compare no regularizer, activation L2, product clipping, and this penalty at equal FLOPs. Track validation accuracy, calibration, activation-tail quantiles, gradient variance, and the fraction of batches with negative eigenvalues. Success is lower activation and gradient outlier rates with equal or better validation performance.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.