# RG Spectral Feature Gate

- ID: 1483
- Canonical URL: https://synthcore.org/idea/1483/rg-spectral-feature-gate
- API JSON: https://synthcore.org/api/idea/1483.json
- API Markdown: https://synthcore.org/api/idea/1483.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.27236](https://arxiv.org/abs/2607.27236)
- Category: architecture
- Solves: accuracy, memory, generalization
- ML areas: embedding, transformer, pruning, regularization
- Math tags: random-matrix, dynamical-systems, probability, spectral-theory, statistics
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Replace fixed PCA-rank selection in a hidden layer with a renormalization-group-inspired gate over covariance eigenvalue bands. The gate retains modes whose effective quartic interaction remains unstable or strongly scale-dependent, while pruning bands that flow toward the Gaussian noise fixed point. Unlike top-eigenvalue truncation, this is designed for extensive-rank signal distributed throughout the bulk spectrum.

## Mathematical statement

The paper defines the empirical correlation matrix C from activations X in R^{N times P} by C_{ij} = (C_0)_{ij} divided by the square root of (C_0)_{ii}(C_0)_{jj}, with covariance C_0 = X^T X divided by N, where N is the number of samples and P is the feature dimension. Pure noise has the Marchenko--Pastur eigenvalue density mu_MP(lambda) = square root of ((lambda_+ - lambda)(lambda - lambda_-)) divided by (2 pi sigma^2 q lambda), where q = P/N, noise variance is sigma^2, and lambda_+ and lambda_- equal sigma^2(1 plus or minus square root of q)^2. The field is expanded in the correlation eigenbasis as phi_i = sum_lambda u_i^(lambda) varphi_lambda, and the retained modes are coupled through the quartic interaction V(phi) = u_4 sum_i phi_i^4 divided by 4!. The paper's detection rule uses the scale-dependent canonical dimension of u_4: the critical scale lambda_c satisfies dim_tau(u_4) evaluated at lambda_c equals zero, while an optimal scale is a local minimum satisfying the derivative of dim_tau(u_4) with respect to beta equals zero. We use spectral scale tau_r = log(lambda_r + epsilon), estimate a bandwise quartic coefficient u_{4,r} from excess fourth cumulants, and use its finite-difference logarithmic slope as a practical proxy for the canonical dimension.

## Key formulas

- $$C_{ij}=\frac{(C_0)_{ij}}{\sqrt{(C_0)_{ii}(C_0)_{jj}}},\qquad C_0=\frac{X^{\mathsf T}X}{N}.$$
- $$\mu_{\mathrm{MP}}(\lambda)=\frac{\sqrt{(\lambda_+-\lambda)(\lambda-\lambda_-)}}{2\pi\sigma^2q\lambda},\qquad \lambda_{\pm}=\sigma^2(1\pm\sqrt q)^2,\quad q=\frac{P}{N}.$$
- $$\phi_i=\sum_{\lambda}u_i^{(\lambda)}\varphi_{\lambda},\qquad V(\phi)=\frac{u_4}{4!}\sum_i\phi_i^4.$$
- $$\widehat d_r=\frac{\log(|u_{4,r+1}|+\epsilon)-\log(|u_{4,r}|+\epsilon)}{\log(\lambda_{r+1}+\epsilon)-\log(\lambda_r+\epsilon)},\qquad \widehat d_{r_c}\approx 0.$$

## Implementation notes

1. Integration point: apply the gate to the feature dimension of a transformer MLP output, convolutional channel activation, or embedding table before the next linear projection. During calibration, collect a rolling matrix X of centered activation vectors from N samples, compute the correlation matrix C, and eigendecompose C = U diag(lambda_1,...,lambda_P) U^T. Do not simply keep the largest eigenvalues. 2. Pseudocode: `X <- centered activation batch; C <- corr(X); lambda,U <- eigh(C); form logarithmic or quantile spectral bands B_r; z_r <- X U[:,B_r]; kappa4_r <- mean(z_r^4) - 3 mean(z_r^2)^2; u4_r <- -kappa4_r/(mean(z_r^2)^2 + eps); d_r <- finite_difference(log(abs(u4_r)+eps), log(lambda_band+eps)); rc <- first r with a sign change in d_r that persists for two bands; mask[B_r] <- 1 for bands on the signal side of rc and 0 otherwise; h_next <- W(mask * U z)`. Use a straight-through estimator for the binary mask during training, or freeze the mask after calibration. 3. Compute the MP edges from q = P/N and estimate sigma^2 from the median bulk eigenvalue. Estimate u4_r and the crossing from minibatches, using an exponential moving average and bootstrap resampling to prevent unstable gates. 4. First experiment: use a 6-layer transformer on CIFAR-10 or a small language-model benchmark, insert the gate after every MLP, and compare dense training, fixed top-k PCA, and magnitude pruning at equal retained width. Measure validation accuracy, retained FLOPs, and mask stability across batches. Success means higher accuracy than top-k PCA at the same width, or at least 20 percent lower width at matched accuracy, with a persistent spectral crossing rather than a validation-tuned cutoff.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built and numerically checked an RG spectral feature-gate MVP with correlation eigendecomposition, bandwise excess-kurtosis estimates, logarithmic slope scoring, and a synthetic downstream comparison. Noise scaling matched N^-1/2 (-0.523 observed vs -0.5 predicted), and |u4| grew approximately linearly with non-Gaussian strength, but low-exponent slope recovery was biased and the RG gate reached 0.525 accuracy versus 0.957 for top-PCA at equal retained width. Therefore the proposed mechanism and practical win were not established.

### Mechanism check

- Verdict: Built and numerically checked an RG spectral feature-gate MVP with correlation eigendecomposition, bandwise excess-kurtosis estimates, logarithmic slope scoring, and a synthetic downstream comparison. Noise scaling matched N^-1/2 (-0.523 observed vs -0.5 predicted), and |u4| grew approximately linearly with non-Gaussian strength, but low-exponent slope recovery was biased and the RG gate reached 0.525 accuracy versus 0.957 for top-PCA at equal retained width. Therefore the proposed mechanism and practical win were not established.
- Confidence: 9/10
- Limitations: Only a synthetic NumPy experiment was tested. No transformer, CIFAR-10, training-time straight-through gate, bootstrap/EMA calibration, FLOP or memory measurement, or minibatch mask-stability test was performed; the practical gate is a heuristic rather than the paper's full canonical-dimension flow.

## Artifacts

- [report.md](https://synthcore.org/code/579/report.md)
- [results.json](https://synthcore.org/code/579/results.json)
- [rg_gate_experiment.py](https://synthcore.org/code/579/rg_gate_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/579)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.