# Rank-one Gaussian intensity feature layer

- ID: 59
- Canonical URL: https://synthcore.org/idea/59/rank-one-gaussian-intensity-feature-layer
- API JSON: https://synthcore.org/api/idea/59.json
- API Markdown: https://synthcore.org/api/idea/59.md
- Verification status: unverified
- Source: [arXiv:2608.22418](https://arxiv.org/abs/2608.22418)
- Category: architecture
- Solves: memory, stability, scalability
- ML areas: embedding, memory, inference-speedup, architecture
- Math tags: linear-algebra, random-matrix, metric-geometry
- Ratings: usefulness 6/10; difficulty 3/10; novelty 7/10

## Idea description

Use fixed or lightly trainable rank-one Gaussian PSD measurements as a compact feature layer for representations whose useful information is contained in a low-rank Gram or covariance matrix. The nonnegative intensity coordinates preserve the geometry of low-rank PSD inputs while using only \(m\) scalar features instead of a full \(d\times d\) matrix.

## Mathematical statement

The paper's rank-one construction sets \(A_i=a_i a_i^*\), with sensing vector \(a_i\in\mathbb H^n\), and obtains \(\langle A_i,xx^*\rangle=|a_i^*x|^2\). More generally, for \(X=HH^*\), the same measurement is \(\langle a_i a_i^*,HH^*\rangle=\|a_i^*H\|_2^2\), which is nonnegative and computable without materializing \(X\). The paper studies \(\beta_{\mathcal A}(r)=U_{\mathcal A}(r)/L_{\mathcal A}(r)\) and reports that random rank-one Gaussian measurements are asymptotically sharp for the universal lower bounds, making them near-optimal for low-rank PSD stability. Here \(d\) is feature dimension, \(s\) is the number of tokens or samples in \(H\), \(m\) is compressed width, and \(a_i\sim\mathcal N(0,I_d/d)\) are independent Gaussian sensing vectors.

## Key formulas

- $$A_i=\bm a_i\bm a_i^*,\qquad \langle A_i,xx^*\rangle=\operatorname{Tr}(\bm a_i\bm a_i^*xx^*)=|\bm a_i^*\bm x|^2.$$
- $$X=HH^*,\qquad \Phi_{\mathcal A}(X)_i=\langle a_i a_i^*,HH^*\rangle=\|a_i^*H\|_2^2\ge0.$$
- $$\beta_{\mathcal A}(r)=\frac{U_{\mathcal A}(r)}{L_{\mathcal A}(r)},\qquad \mathcal A=\{a_i a_i^*\}_{i=1}^{m},\quad a_i\overset{\mathrm{i.i.d.}}{\sim}\mathcal N(0,I_d/d).$$

## Implementation notes

Integrate the layer after a token encoder and before a memory-heavy attention, pooling, or retrieval stage. Given token features \(H\in\mathbb R^{d\times s}\), sample one fixed matrix \(A\in\mathbb R^{m\times d}\) at initialization with entries \(A_{ik}\sim\mathcal N(0,1/d)\), and output \(z\in\mathbb R^m\) with `z=(A@H).square().mean(dim=1)`. This is exactly the rank-one PSD map because each row \(a_i^\top\) defines \(A_i=a_i a_i^\top\); do not form the \(d\times d\) Gram matrix. Optionally add a trainable positive scale \(g_i=\operatorname{softplus}(u_i)\) and use `z=g*z`, while keeping Gaussian directions frozen to isolate the mathematical design. For a decoder needing signed information, concatenate a conventional low-dimensional linear projection or apply this layer only to covariance-invariant branches, since intensity features do not preserve the sign of \(H\). Start with \(m\in\{d/4,d/2,d\}\), rank-controlled synthetic data \(H=UV^\top\) with rank \(r\in\{2,4,8\}\), and a CIFAR-10 or small-language-model feature-compression task. Compare against dense random projection, learned linear projection, and explicit Gram-vectorization followed by PCA. Record activation memory, wall-clock throughput, reconstruction or task error, and pairwise distortion \(q(X,Y)=\|\Phi(X)-\Phi(Y)\|_1/\|X-Y\|_F\). The target is comparable task accuracy with reduced covariance storage and stable distortion while \(r\) remains below the design rank.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.