# Completely-positive bilinear covariance layer

- ID: 846
- Canonical URL: https://synthcore.org/idea/846/completely-positive-bilinear-covariance-layer
- API JSON: https://synthcore.org/api/idea/846.json
- API Markdown: https://synthcore.org/api/idea/846.md
- Verification status: unverified
- Source: [arXiv:2607.13251](https://arxiv.org/abs/2607.13251)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: mlp, graph-nn, attention
- Math tags: linear-algebra, convex-analysis, optimization
- Ratings: usefulness 5/10; difficulty 5/10; novelty 5/10

## Idea description

Replace an unconstrained bilinear matrix fusion or covariance head with \(\Phi(A,B)=\sum_{r=1}^R V_r^*(A\otimes B)V_r\). The output is PSD by construction, and the stronger block-level property makes the layer compatible with minibatches, mixtures, and Gram-matrix inputs rather than merely preserving positivity pointwise.

## Mathematical statement

The paper defines a bilinear map \(\Phi:M_n(\mathbb C)\times M_n(\mathbb C)\to M_m(\mathbb C)\), where \(M_n(\mathbb C)\) is the space of \(n\times n\) complex matrices. It is jointly completely positive when, for every integer \(\ell\ge 1\) and every positive-semidefinite block matrices \(A=(A_{i,j})_{i,j=0}^{\ell-1}\) and \(B=(B_{s,t})_{s,t=0}^{\ell-1}\) in \(M_{n\ell}(\mathbb C)\), the block matrix \(\big((\Phi(A_{i,j},B_{s,t})_{s,t=0}^{\ell-1})\big)_{i,j=0}^{\ell-1}\) is positive semidefinite. The paper states that this is equivalent to complete positivity of the associated linear map on the tensor product and equivalent to a Choi matrix being positive semidefinite. Most directly for implementation, it is equivalent to the Choi-Kraus representation \(\Phi(A,B)=\sum_{r=1}^{R}V_r^*(A\otimes B)V_r\), where \(A\otimes B\in M_{n^2}(\mathbb C)\), each \(V_r\in\mathbb C^{n^2\times m}\), \(V_r^*\) is the conjugate transpose, and \(R\) is the chosen Kraus rank. For Hermitian PSD \(A,B\), every term \(V_r^*(A\otimes B)V_r\) is PSD, so their sum is PSD. In a real neural network, use transpose instead of conjugate transpose and parameterize \(V_r\) as trainable real matrices.

## Key formulas

- $$0\preceq\big((\Phi(A_{i,j},B_{s,t})_{s,t=0}^{\ell-1})\big)_{i,j=0}^{\ell-1}$$
- $$\Phi(A,B)=\sum_{r=1}^{R}V_r^{*}(A\otimes B)V_r$$
- $$\Phi_{\theta}(A,B)=\epsilon I_m+\sum_{r=1}^{R}V_r^{\top}(A\otimes B)V_r,\qquad A\succeq0,\;B\succeq0,\;\epsilon\ge0$$
- $$\mathcal L_{\mathrm{cov}}=-\log\det(\Phi_{\theta}(A,B))-y^{\top}\Phi_{\theta}(A,B)^{-1}y$$

## Implementation notes

Integrate this at a covariance, kernel, or matrix-valued fusion head. The inputs are two PSD matrices \(A,B\in\mathbb R^{n\times n}\), obtained for example as \(A=XX^{\top}+\delta I_n\) and \(B=YY^{\top}+\delta I_n\) from two token or modality embeddings; the output is an \(m\times m\) covariance or Gram matrix. Store \(R\) trainable matrices \(V[r]\in\mathbb R^{n^2\times m}\), and use a small fixed \(\epsilon>0\) for numerical conditioning. Pseudocode: `A = X @ X.T + delta*I; B = Y @ Y.T + delta*I; K = kron(A,B); C = epsilon*I_m; for r in range(R): Z = K @ V[r]; C += V[r].T @ Z; C = 0.5*(C+C.T)`. Feed \(C\) to a Gaussian negative-log-likelihood, Mahalanobis loss, PSD attention kernel, or graph diffusion module. Do not materialize the Kronecker matrix for large \(n\): reshape \(V_r\) into an \(n\times n\times m\) tensor and compute \((A\otimes B)V_r\) by two successive contractions, or use a Kronecker-factored parameterization \(V_r=U_r\otimes W_r\) as a cheaper ablation. The PSD guarantee is computed analytically from the Kraus form; estimate empirical conditioning via the smallest eigenvalue of \(C\), and estimate compute and memory from the chosen \(R\). First experiment: on a small heteroscedastic regression or graph-link-prediction task, compare an unconstrained bilinear head, a Cholesky covariance head, and this layer at matched parameter counts. Track validation NLL, calibration, fraction of batches with negative eigenvalues, minimum eigenvalue, wall-clock time, and gradient-norm spikes. Success is zero PSD violations, fewer unstable training steps, and lower NLL or better calibration at comparable FLOPs; a useful negative result is no accuracy gain after accounting for the extra contraction cost.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.