Lindeberg-Regularized Poisson Random Features

(1) Integration point: implement this as a replacement for a Gaussian random-feature projection or as an optional sparse MLP or embedding projection. Discretize the feature domain into \(m\) atoms with Poisson rates \(\lambda_j\). For an MVP, use order \(q=2\), a symmetric kernel matrix \(K\in\mathbb R^{m\times m}\) with zero diagonal, and centered counts \(\xi_j=(N_j-\lambda_j)/\sqrt{\lambda_j}\), where \(N_j\sim\operatorname{Poisson}(\lambda_j)\). For input \(h\), use a learned input-dependent kernel \(K(h)\), or first project \(h\) to coefficients \(a_j(h)\); form \(F(h)=\sum_{j\ne k}K_{jk}(h)\xi_j\xi_k\). Form the Gaussian control feature \(G(h)=\sum_{j\ne k}K_{jk}(h)z_jz_k\), with \(z_j\sim N(0,1)\), using the same kernel. (2) Pseudocode: sample counts \(N\), compute \(\xi\), and compute \(F\). For each selected atom \(r\), evaluate an add-one value by replacing \(N_r\leftarrow N_r+1\), recomputing \(F^{(+r)}\), and setting \(D_rF=F^{(+r)}-F\). Accumulate the minibatch importance-sampling estimate \(\widehat{\mathcal E}_4=\frac1B\sum_{b=1}^B\sum_{r\in S_b}\mu_r|D_rF_b|^4/p_r\), where \(S_b\) is a sampled atom subset, \(p_r\) is its inclusion probability, and \(\mu_r\) is the atom intensity. Add the normalized penalty and variance penalty to the task loss, then backpropagate. (3) The paper-derived object is the fourth add-one energy and its vanishing-energy invariance interpretation. The minibatch variance, atom subsampling, inclusion probabilities, and differentiable finite-difference estimator are engineering choices. Use common random numbers when comparing Poisson and Gaussian features so both use identical kernels and inputs. (4) First cheap experiment: train a two-layer MLP on MNIST with a 512-feature Gaussian random projection as baseline. Compare it with a Poisson order-2 layer at equal expected active-feature count, both without regularization and with the add-one penalty. Track validation accuracy, gradient-norm variance, activation kurtosis, empirical Wasserstein distance between Poisson and Gaussian outputs, and wall-clock cost. Also test a rare-jump regime with small atom rates; the hypothesis is that add-one regularization improves stability and accuracy there, reducing activation tails and Poisson-versus-Gaussian discrepancy at equal sparsity. A negative result is equally clear: if the penalty does not improve accuracy or gradient stability at fixed expected active count, discard the method.