# Distributional spectral-preconditioned features

- ID: 1585
- Canonical URL: https://synthcore.org/idea/1585/distributional-spectral-preconditioned-features
- API JSON: https://synthcore.org/api/idea/1585.json
- API Markdown: https://synthcore.org/api/idea/1585.md
- Verification status: unverified
- Source: [arXiv:2607.29258](https://arxiv.org/abs/2607.29258)
- Category: architecture
- Solves: stability, accuracy, sample-efficiency
- ML areas: mlp, initialization, regularization, training-dynamics
- Math tags: harmonic-analysis, functional-analysis, measure-theory
- Ratings: usefulness 5/10; difficulty 5/10; novelty 8/10

## Idea description

Replace or augment a singular scalar activation \(\sigma\) with a distributionally regularized activation \(g\) whose Fourier transform is multiplied by \((i\rho)^\alpha\). This suppresses the problematic low-frequency singular component and can produce better-conditioned random-feature or first-layer representations, while a residual raw-activation branch prevents loss of standard approximation behavior.

## Mathematical statement

The paper defines the regularized spectrum by \(\widehat{g}(\rho)=(i\rho)^\alpha\widehat{\sigma}(\rho)\), where \(\sigma:\mathbb{R}\to\mathbb{R}\) is the possibly distributional activation, \(\widehat{\sigma}\) is its one-dimensional Fourier transform, \(\rho\in\mathbb{R}\) is frequency, and \(\alpha\) is the nonnegative regularization or derivative order. For integer \(\alpha\), the spatial-domain object is the distributional derivative \(g=D^\alpha\sigma\); for noninteger \(\alpha\), it is the Fourier multiplier with symbol \((i\rho)^\alpha\), using a fixed branch convention. A ridge feature is \(\phi_{w,b}(x)=g(w^\top x+b)\), where \(x\in\mathbb{R}^d\), \(w\in\mathbb{R}^d\), and \(b\in\mathbb{R}\). The practical adaptation uses \(\phi^{\mathrm{mix}}_{w,b}(x)=\phi_{w,b}(x)+\beta\sigma(w^\top x+b)\), with residual coefficient \(\beta\). The paper's key property is that the multiplier absorbs the origin singularity into a standard tempered-distribution representation with improved spatial decay; the ML hypothesis is that this reduces feature-covariance ill-conditioning.

## Key formulas

- $$\widehat{g}(\rho)=(i\rho)^{\alpha}\widehat{\sigma}(\rho)$$
- $$g(t)=\mathcal{F}^{-1}\!\left[(i\rho)^\alpha\widehat{\sigma}(\rho)\right](t)=D^\alpha\sigma(t)$$
- $$\phi^{\mathrm{mix}}_{w,b}(x)=D^\alpha\sigma(w^\top x+b)+\beta\,\sigma(w^\top x+b)$$
- $$\min_{a\in\mathbb{R}^m}\frac{1}{n}\|\Phi a-y\|_2^2+\lambda\|a\|_2^2,\qquad \Phi_{j,k}=\phi^{\mathrm{mix}}_{w_k,b_k}(x_j)$$

## Implementation notes

Integrate this at the scalar activation in a shallow MLP or in the first layer of a small residual MLP. Choose a baseline activation with a known singular spectrum, such as ReLU, and begin with integer \(\alpha=1\) or \(2\); defer fractional orders until the integer experiment is stable. Sample fixed random \(w_k\sim\mathcal{N}(0,I_d/d)\) and biases \(b_k\) using the baseline random-feature initialization. For each minibatch, compute \(u=XW+b\), evaluate \(G=D^\alpha\sigma_\varepsilon(u)\), and form \(Z=G+\beta\sigma_\varepsilon(u)\). Feed \(Z\) to a learned linear output head and optimize the ridge objective or its SGD equivalent. Pseudocode: sample \(W,b\); compute \(u\); compute \(G=\text{derivative}(\sigma_\varepsilon,u,\alpha)\); set \(Z=G+\beta\sigma_\varepsilon(u)\); predict \(Za\); backpropagate loss. The mathematical quantity is the Fourier-multiplier definition of \(D^\alpha\); numerically, use autodifferentiation for integer orders or FFT convolution with multiplier \((i\rho)^\alpha\) on a bounded grid. For ReLU, \(D\operatorname{ReLU}(t)=\mathbf{1}_{t>0}\), while higher derivatives require smoothing. Estimate the feature condition number, smallest singular value, gradient variance, optimization loss, and validation accuracy. Start with MNIST or CIFAR-10 and 2,048 fixed features, comparing ReLU, derivative-only, and residual-mixture features at matched feature count and FLOPs. Success is lower condition number and faster loss reduction at equal compute without test-accuracy degradation; failure is conditioning improvement accompanied by worse approximation or generalization.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.