# Gramian-Gated Feature Pruning

- ID: 2520
- Canonical URL: https://synthcore.org/idea/2520/gramian-gated-feature-pruning
- API JSON: https://synthcore.org/api/idea/2520.json
- API Markdown: https://synthcore.org/api/idea/2520.md
- Verification status: unverified
- Source: [arXiv:2608.18380](https://arxiv.org/abs/2608.18380)
- Category: memory
- Solves: memory, stability, speedup
- ML areas: embedding, pruning, inference-speedup
- Math tags: linear-algebra, spectral-theory, numerical-analysis
- Ratings: usefulness 6/10; difficulty 4/10; novelty 5/10

## Idea description

Use the weighted feature Gramian to remove nearly dependent Fourier, sinusoidal, or dictionary features during training instead of relying only on magnitude pruning. The criterion detects redundant directions even when their learned coefficients are individually large, improving conditioning of the linear readout and reducing inference cost.

## Mathematical statement

Let B in C^{N x M} contain the current neural dictionary evaluated on N calibration samples, W be a positive diagonal matrix of sample or boundary weights, and G=B^*WB be the weighted Gramian. Its entry G_{mn}=sum_i w_i conjugate(B_{im})B_{in} is the weighted correlation between features m and n. If the eigenvalues satisfy lambda_1>=...>=lambda_M>=0, select numerical rank r by retaining lambda_j>=tau lambda_1, where tau is a cutoff such as 10^{-6} in float32 or 10^{-10} in float64. The retained feature subspace is span{v_j:lambda_j>=tau lambda_1}. This is the neural analogue of the paper's local Cauchy-trace Gramian used to remove numerically dependent directions. A ridge-stabilized readout uses G_lambda=G+lambda I, with lambda>0 preventing unstable solves when the retained dictionary is only approximately independent.

## Key formulas

- $$G=B^*WB,\qquad G_{mn}=\sum_{i=1}^{N}w_i\,\overline{B_{im}}B_{in}.$$
- $$r=\max\{j:\lambda_j(G)\geq\tau\lambda_1(G)\},\qquad \mathcal{S}_{\mathrm{keep}}=\operatorname{span}\{v_j:\lambda_j(G)\geq\tau\lambda_1(G)\}.$$
- $$c=(G+\lambda I)^{-1}B^*Wy,\qquad \lambda>0.$$

## Implementation notes

Attach the procedure to any Fourier-feature coordinate network, random-feature readout, low-rank neural operator, or mixture-of-experts router whose candidate features can be evaluated on a calibration batch. The exact integration point is after feature construction and before the linear projection or final readout. Every K training steps, build B on N representative coordinates, set W=diag(w_1,...,w_N) using uniform weights initially, and compute G=B.conj().T@W@B. Normalize each column of B first if feature amplitudes differ substantially. Compute the eigendecomposition of the Hermitian G; retain eigenvectors above tau times the largest eigenvalue, or greedily remove the feature with largest diagonal redundancy until the condition number of the retained Gramian is below a target kappa. Refit the readout coefficients after pruning by solving (G+lambda I)c=B.conj().T@W@y. For a nonlinear downstream MLP, either physically delete the corresponding channels and project the next-layer weights, or apply binary gates with straight-through gradients and permanently remove gates after validation. Math-computed quantities are G, its spectrum, numerical rank, and the ridge solve; tau, K, lambda, and calibration-sample selection are empirical. First test on a 2D regression task y(x)=sum_{m=1}^{32}a_m cos(k_m^T x) with duplicated and near-duplicated candidate directions, comparing magnitude pruning and no pruning. Track condition number, retained width, validation error, and inference latency. Success means at least 30% feature removal with unchanged error and a substantial reduction in solve or inference time; instability or accuracy collapse under small coordinate perturbations falsifies the approach.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.