# Polynomial Band-Pass Feature Mixer

- ID: 134
- Canonical URL: https://synthcore.org/idea/134/polynomial-band-pass-feature-mixer
- API JSON: https://synthcore.org/api/idea/134.json
- API Markdown: https://synthcore.org/api/idea/134.md
- Verification status: unverified
- Source: [arXiv:2608.24444](https://arxiv.org/abs/2608.24444)
- Category: architecture
- Solves: accuracy, stability
- ML areas: transformer, mlp, regularization, training-dynamics
- Math tags: spectral-theory, linear-algebra, dynamical-systems, geometry
- Ratings: usefulness 5/10; difficulty 5/10; novelty 6/10

## Idea description

Add a norm-controlled feature mixer that applies a polynomial spectral filter to the channel covariance of a transformer or MLP block. A quadratic filter centered at \(\rho\) suppresses covariance eigenmodes far from the target and preserves modes near it, providing a tunable alternative to purely variance-maximizing mixing or standard normalization.

## Mathematical statement

For a positive self-adjoint operator \(\mathcal{B}\) with eigenpairs \((e_j,\lambda_j)\), the paper applies \(e^{-t\mathscr{P}(\mathcal{B})}\) and renormalizes. Active spectral support is preserved, while components with larger polynomial values decay faster; for \(\mathscr{P}(s)=(s-\rho)^2\), eigenvalues nearest \(\rho\) are selected, including ties when two levels are equally distant. Adapt this to a batch feature matrix \(X\in\mathbb{R}^{n\times d}\), with channel covariance \(B=X^TX/n+\epsilon I_d\), eigenvalues \(\lambda_j\geq0\), and right-side filtering by \(\mathscr{P}(B)\). The sphere normalization is applied to the filtered feature tensor to prevent uncontrolled scale changes.

## Key formulas

- $$\mathcal{B}e_j=\lambda_je_j,\qquad \mathscr{P}(\mathcal{B})e_j=\mathscr{P}(\lambda_j)e_j,$$
- $$X_{\mathrm{filt}}=\frac{Xe^{-t\mathscr{P}(B)}}{\|Xe^{-t\mathscr{P}(B)}\|_F},\qquad B=\frac{1}{n}X^TX+\epsilon I,$$
- $$\mathscr{P}(s)=(s-\rho)^2,\qquad \mathscr{P}(B)X^T=B^2X^T-2\rho BX^T+\rho^2X^T,$$
- $$Y_{k+1}=\frac{Y_k-\eta\left(Y_kB^2-2\rho Y_kB+\rho^2Y_k\right)}{\left\|Y_k-\eta(Y_kB^2-2\rho Y_kB+\rho^2Y_k)\right\|_F}.$$

## Implementation notes

Insert the module after the projection output and before the residual addition in a transformer block, or after the hidden activation in an MLP. For token features \(X\in\mathbb{R}^{n\times d}\), center across tokens with \(X\leftarrow X-\operatorname{mean}_n(X)\), compute \(B=X^TX/n+\epsilon I_d\), and detach \(B\) initially to avoid excessive second-order memory. Use a residual version: set \(Y_0=X\), run \(K=1\) to \(3\) explicit Euler steps, compute \(YB=YB\), \(YB^2=(YB)B\), form \(Q=YB^2-2\rho YB+\rho^2Y\), and update \(Y\leftarrow Y-\eta Q\). Normalize only the filtered branch by per-token RMS, then output \(X+\alpha(\operatorname{RMSNorm}(Y)-X)\), with learnable \(\alpha\) initialized to 0.1. Estimate \(\rho\) using \(\rho=\operatorname{tr}(B)/d\), or learn a positive scalar parameterized by softplus. The paper contributes the spectral interpretation: in the eigenbasis of \(B\), each mode is multiplied by \(e^{-t(\lambda_j-\rho)^2}\), so modes far from \(\rho\) are attenuated. Start with a 6-layer transformer on CIFAR-10 or WikiText-2, comparing the module against LayerNorm, RMSNorm, and a standard low-pass polynomial filter. Track validation accuracy or perplexity, covariance eigenvalue histograms, activation RMS, rank, and gradient variance. The desired signal is reduced activation instability and improved validation performance at equal parameter count; reject the method if it causes rank collapse or adds wall-clock cost without accuracy or stability gains.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.