De-floored low-rank feature preconditioner
Implementation & benchmark of arXiv:2607.16638 — De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction
Source paper: De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction arXiv:2607.16638 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace the usual inverse-eigenvalue weights in a low-rank feature-covariance preconditioner by inverse weights with an estimated isotropic floor subtracted. Retain only the top r eigendirections and require every corrected denominator to exceed a margin, preventing the shifted inverse from approaching a pole. This should undo systematic under-updating of predictive directions when many weak feature directions inflate the empirical covariance.
Formulas
Mathematical statement
The paper's structural calculation is $q_{\mathrm{PCR},j}=s_j/(s_j+a)$, where $s_j$ is the realized scale of predictive direction $j$ and $a$ is the scalar covariance floor contributed by the high-rank tail; ordinary spectral normalization divides by $s_j+a$ and therefore leaves a multiplicative attenuation even when the rank is selected correctly. dPCR instead uses empirical eigenpairs $(\widehat{\mu}_i,\widehat{v}_i)$ and replaces the denominator $\widehat{\mu}_i$ by $\widehat{\mu}_i-a_0$, where $a_0$ estimates the isotropic tail floor. For a neural feature covariance $C\in\mathbb{R}^{d\times d}$, let $\widehat{\mu}_1\geq\cdots\geq\widehat{\mu}_d\geq0$ and orthonormal eigenvectors $\widehat v_i$ satisfy $C\widehat v_i=\widehat\mu_i\widehat v_i$. Estimate the floor at rank $r$ by a trimmed mean of the non-retained eigenvalues, $\widehat a_0=\operatorname{mean}\{\widehat\mu_i:i\in T\}$, where $T\subseteq\{r+1,\ldots,d\}$ discards a fixed fraction of the largest and smallest tail eigenvalues. Use the corrected low-rank operator $P_r=\sum_{i=1}^r(\widehat\mu_i-\widehat a_0)^{-1}\widehat v_i\widehat v_i^\top$, provided $\widehat\mu_r-\widehat a_0\geq\delta>0$. To preserve updates outside the learned subspace, combine it with a scalar complement: $P_{r,\eta}=P_r+\eta^{-1}(I-\sum_{i=1}^r\widehat v_i\widehat v_i^\top)$. The key prediction-side mechanism is that subtracting $a_0$ restores the intended multiplier from $s_j/(s_j+a)$ toward one, while the positive margin controls stability.
Implementation notes
1. Integration point: apply this to the input-feature side of a trainable linear layer, especially the final classifier or an MLP projection. For a layer with activations H in a minibatch, form a periodically refreshed covariance C=\beta C+(1-\beta)H^ op H/B, where B is batch size and H is centered or RMS-normalized. Precondition the layer gradient G_W on the right by P_{r,\eta}; do not alter the forward pass. 2. Pseudocode: every K optimizer steps, compute the top r eigenpairs (mu, V)=Lanczos(C,r); collect tail eigenvalues from a cheap additional spectrum estimate or from a randomized range complement; discard the highest and lowest q fraction of the tail and set a0=mean(trimmed_tail). Set a0=min(a0, mu[r]-delta) to guarantee the denominator margin. For every update, form P action without materializing a d-by-d matrix: Y=G_W V; Y=Y/(mu-a0) elementwise; Gcorr=Y V^T + (G_W-G_W V V^T)/eta; update W <- W-alpha Gcorr, optionally with Adam moments applied before or after this preconditioning. 3. Computed from the paper's mathematics: the shifted inverse, hard rank cutoff, floor estimate, and margin condition. Estimated empirically: the covariance spectrum, r, beta, trimming fraction q, delta, and complement scale eta. Use exponential covariance averaging to reduce minibatch noise and clip a0 at zero if the estimated tail is not positive. 4. First experiment: train a width-512 two-layer MLP on CIFAR-10 or MNIST with a 256-dimensional penultimate representation and a linear classifier. Compare SGD, AdamW, ordinary rank-r covariance preconditioning using 1/mu_i, and the de-floored version at equal matrix-vector-product cost, sweeping r in {8,16,32} and K in {10,50}. Log eigenvalues, estimated floor, denominator margins, training loss, validation accuracy, and gradient norms. The expected signal is faster loss descent or higher accuracy at a fixed step budget when tail eigenvalues form a stable floor; the method should lose or become unstable when mu_r is not separated from the estimated floor, providing a direct falsification criterion.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.