Unverified 2026

Polynomial Band-Pass Feature Mixer

Implementation & benchmark of arXiv:2608.24444 — Spectral Selection in Sphere-Constrained Flows Generated by Polynomials of the Dirichlet Laplacian

Usefulness5/10

Difficulty5/10

Novelty6/10

Architecture Accuracy Stability Transformer Mlp Regularization Training dynamics Spectral theory Linear algebra Dynamical systems Geometry

Source paper: Spectral Selection in Sphere-Constrained Flows Generated by Polynomials of the Dirichlet Laplacian arXiv:2608.24444 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add a norm-controlled feature mixer that applies a polynomial spectral filter to the channel covariance of a transformer or MLP block. A quadratic filter centered at $\rho$ suppresses covariance eigenmodes far from the target and preserves modes near it, providing a tunable alternative to purely variance-maximizing mixing or standard normalization.

Formulas

$$\mathcal{B}e_j=\lambda_je_j,\qquad \mathscr{P}(\mathcal{B})e_j=\mathscr{P}(\lambda_j)e_j,$$

$$X_{\mathrm{filt}}=\frac{Xe^{-t\mathscr{P}(B)}}{\|Xe^{-t\mathscr{P}(B)}\|_F},\qquad B=\frac{1}{n}X^TX+\epsilon I,$$

$$\mathscr{P}(s)=(s-\rho)^2,\qquad \mathscr{P}(B)X^T=B^2X^T-2\rho BX^T+\rho^2X^T,$$

$$Y_{k+1}=\frac{Y_k-\eta\left(Y_kB^2-2\rho Y_kB+\rho^2Y_k\right)}{\left\|Y_k-\eta(Y_kB^2-2\rho Y_kB+\rho^2Y_k)\right\|_F}.$$

Mathematical statement

For a positive self-adjoint operator $\mathcal{B}$ with eigenpairs $(e_j,\lambda_j)$, the paper applies $e^{-t\mathscr{P}(\mathcal{B})}$ and renormalizes. Active spectral support is preserved, while components with larger polynomial values decay faster; for $\mathscr{P}(s)=(s-\rho)^2$, eigenvalues nearest $\rho$ are selected, including ties when two levels are equally distant. Adapt this to a batch feature matrix $X\in\mathbb{R}^{n\times d}$, with channel covariance $B=X^TX/n+\epsilon I_d$, eigenvalues $\lambda_j\geq0$, and right-side filtering by $\mathscr{P}(B)$. The sphere normalization is applied to the filtered feature tensor to prevent uncontrolled scale changes.

Implementation notes

Insert the module after the projection output and before the residual addition in a transformer block, or after the hidden activation in an MLP. For token features $X\in\mathbb{R}^{n\times d}$, center across tokens with $X\leftarrow X-\operatorname{mean}_n(X)$, compute $B=X^TX/n+\epsilon I_d$, and detach $B$ initially to avoid excessive second-order memory. Use a residual version: set $Y_0=X$, run $K=1$ to $3$ explicit Euler steps, compute $YB=YB$, $YB^2=(YB)B$, form $Q=YB^2-2\rho YB+\rho^2Y$, and update $Y\leftarrow Y-\eta Q$. Normalize only the filtered branch by per-token RMS, then output $X+\alpha(\operatorname{RMSNorm}(Y)-X)$, with learnable $\alpha$ initialized to 0.1. Estimate $\rho$ using $\rho=\operatorname{tr}(B)/d$, or learn a positive scalar parameterized by softplus. The paper contributes the spectral interpretation: in the eigenbasis of $B$, each mode is multiplied by $e^{-t(\lambda_j-\rho)^2}$, so modes far from $\rho$ are attenuated. Start with a 6-layer transformer on CIFAR-10 or WikiText-2, comparing the module against LayerNorm, RMSNorm, and a standard low-pass polynomial filter. Track validation accuracy or perplexity, covariance eigenvalue histograms, activation RMS, rank, and gradient variance. The desired signal is reduced activation instability and improved validation performance at equal parameter count; reject the method if it causes rank collapse or adds wall-clock cost without accuracy or stability gains.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.