Distributional spectral-preconditioned features

Integrate this at the scalar activation in a shallow MLP or in the first layer of a small residual MLP. Choose a baseline activation with a known singular spectrum, such as ReLU, and begin with integer \(\alpha=1\) or \(2\); defer fractional orders until the integer experiment is stable. Sample fixed random \(w_k\sim\mathcal{N}(0,I_d/d)\) and biases \(b_k\) using the baseline random-feature initialization. For each minibatch, compute \(u=XW+b\), evaluate \(G=D^\alpha\sigma_\varepsilon(u)\), and form \(Z=G+\beta\sigma_\varepsilon(u)\). Feed \(Z\) to a learned linear output head and optimize the ridge objective or its SGD equivalent. Pseudocode: sample \(W,b\); compute \(u\); compute \(G=\text{derivative}(\sigma_\varepsilon,u,\alpha)\); set \(Z=G+\beta\sigma_\varepsilon(u)\); predict \(Za\); backpropagate loss. The mathematical quantity is the Fourier-multiplier definition of \(D^\alpha\); numerically, use autodifferentiation for integer orders or FFT convolution with multiplier \((i\rho)^\alpha\) on a bounded grid. For ReLU, \(D\operatorname{ReLU}(t)=\mathbf{1}_{t>0}\), while higher derivatives require smoothing. Estimate the feature condition number, smallest singular value, gradient variance, optimization loss, and validation accuracy. Start with MNIST or CIFAR-10 and 2,048 fixed features, comparing ReLU, derivative-only, and residual-mixture features at matched feature count and FLOPs. Success is lower condition number and faster loss reduction at equal compute without test-accuracy degradation; failure is conditioning improvement accompanied by worse approximation or generalization.