Unverified 2026

PSD-Relaxed Multiplicative Gate

Implementation & benchmark of arXiv:2608.16836 — Nonnegative Quadratics over a Quadrant with a Bilinear Constraint

Usefulness6/10

Difficulty7/10

Novelty8/10

Architecture Stability Accuracy Generalization Mlp Attention Moe routing Convex analysis Optimization Linear algebra

Source paper: Nonnegative Quadratics over a Quadrant with a Bilinear Constraint arXiv:2608.16836 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace an unconstrained multiplicative interaction between two nonnegative neural features by a lifted gate whose first and second moments satisfy the paper's semidefinite relaxation for the set F = {(x1,x2): x1,x2 >= 0, x1 x2 <= 1}. Insert the gate into an MLP, attention score, or MoE router to prevent explosive feature products while retaining a tractable convex feasible set.

Formulas

$$F=\{(x_1,x_2)\in\mathbb{R}^{2}:x_1\geq0,\ x_2\geq0,\ x_1x_2\leq1\}$$

$$Y=\begin{pmatrix}1&x_1&x_2\\x_1&X_{11}&X_{12}\\x_2&X_{12}&X_{22}\end{pmatrix},\qquad 0\leq X_{12}\leq1,\quad x_1,x_2\geq0,\quad X_{11}\geq x_1^2,\quad x_1x_2\geq X_{12}^2$$

$$\begin{pmatrix}x_2&X_{12}\\X_{12}&s_1\end{pmatrix}\succeq0,\qquad\begin{pmatrix}X_{11}&s_1&x_1-s_1\\s_1&X_{12}&0\\x_1-s_1&0&1-X_{12}\end{pmatrix}\succeq0$$

$$\begin{pmatrix}x_1&X_{12}\\X_{12}&s_2\end{pmatrix}\succeq0,\qquad\begin{pmatrix}X_{22}&s_2&x_2-s_2\\s_2&X_{12}&0\\x_2-s_2&0&1-X_{12}\end{pmatrix}\succeq0$$

Mathematical statement

The paper studies F = {(x1,x2) in R^2: x1 >= 0, x2 >= 0, x1 x2 <= 1} and represents a lifted point by Y = [[1,x1,x2],[x1,X11,X12],[x2,X12,X22]], where x1 and x2 are first moments and X11, X12, X22 are second moments. Theorem 2.4 introduces shadow variables s1 and s2 for cubic moments x1^2 x2 and x1 x2^2. The required constraints are M1 = [[x2,X12],[X12,s1]] >= 0, M2 = [[x1,X12],[X12,s2]] >= 0, M3 = [[X11,s1,x1-s1],[s1,X12,0],[x1-s1,0,1-X12]] >= 0, and M4 = [[X22,s2,x2-s2],[s2,X12,0],[x2-s2,0,1-X12]] >= 0. Basic Shor-RLT consequences are 0 <= X12 <= 1, x1,x2 >= 0, X11 >= x1^2, and x1 x2 >= X12^2. The transferable property is that nonlinear product constraints are represented by four constant-size PSD constraints.

Implementation notes

Integrate this as a constrained multiplicative feature block rather than constraining every hidden neuron. Given nonnegative feature vectors a and b, produce a = softplus(u)/c and b = softplus(v)/c so typical coordinates are O(1). For each selected channel pair, introduce lifted variables X11, X12, X22, s1, and s2 and define the interaction output g = X12; use h = W1 a + W2 b + W3 g or h = g * value. During training, solve a projection problem min_z ||z-z_raw||^2 over z = (X11,X12,X22,s1,s2) subject to the four PSD constraints, with x1=a and x2=b. For an MVP, use projected gradient: eigendecompose each 2x2 or 3x3 matrix, clip negative eigenvalues to zero, map the corrected matrices back to shared variables, and repeat 3-10 times. A cvxpylayer implementation can provide a reference result. The paper's matrices and constraints are exact; projection iterations and feature scaling are engineering choices. First test a two-layer CIFAR-10 MLP with ordinary multiplication, clipped multiplication, and this gate at equal parameter count. Measure task loss, minimum eigenvalue, gradient norms, activation tails, and accuracy. Success means fewer exploding-gradient events and equal or higher accuracy at the same training budget.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.