Unverified 2026

Dense-support discrete random features

Implementation & benchmark of arXiv:2607.16551 — On high probability of universal approximation in random basis expansions with non-continuous weight sampling

Usefulness5/10

Difficulty4/10

Novelty5/10

Architecture Memory Speedup Accuracy Mlp Quantization Initialization Inference speedup Approximation theory Probability Functional analysis

Source paper: On high probability of universal approximation in random basis expansions with non-continuous weight sampling arXiv:2607.16551 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Build a single-hidden-layer network whose hidden weights and biases are sampled from a non-continuous distribution supported on a dense subset of parameter space, then train only the output coefficients. The result motivates discrete or mixed-precision hidden parameters without requiring a continuous Gaussian initialization; finite-width experiments can test whether this retains accuracy while reducing hidden-layer storage and arithmetic cost.

Formulas

$$\widehat{f}(\boldsymbol{x})=\sum_{j=1}^{m}\alpha_{j}\phi(\boldsymbol{\omega}_{j}^{\top}\boldsymbol{x}+b_{j}),\qquad \boldsymbol{\omega}_{j}\sim\pi,\quad b_j\sim\nu.$$

$$\operatorname{supp}(\pi)\ \text{and}\ \operatorname{supp}(\nu)\ \text{are dense in the corresponding weight and bias spaces}.$$

$$\forall\varepsilon>0,\ \forall\delta>0,\ \exists m_0\ \text{such that}\ m\ge m_0\Longrightarrow\Pr\!\left(\inf_{\alpha_1,\ldots,\alpha_m}\|f-\widehat f\|<\varepsilon\right)\ge 1-\delta.$$

$$q=2^{-r}k,\qquad r\sim p_r,\quad k\sim\operatorname{Unif}\{-K_r,\ldots,K_r\}.$$

Mathematical statement

The random basis expansion is $\widehat{f}(\boldsymbol{x})=\sum_{j=1}^{m}\alpha_j\phi(\boldsymbol{\omega}_j^{\top}\boldsymbol{x}+b_j)$, where $f:\mathbb{C}^{n}\to\mathbb{C}$ is the target, $\boldsymbol{x}\in\mathbb{C}^{n}$ is the input, $m$ is the number of random features, $\alpha_j\in\mathbb{C}$ are trainable output coefficients, $\phi$ is the activation, $\boldsymbol{\omega}_j$ are independently sampled hidden weight vectors from distribution $\pi$, and $b_j$ are independently sampled biases from distribution $\nu$. The paper extends random-basis universal approximation to non-continuous distributions whose supports are dense in the corresponding weight spaces. Informally, for every approximation tolerance $\varepsilon>0$ and failure probability $\delta>0$, sufficiently many independently sampled features satisfy $\inf_{\alpha_1,\ldots,\alpha_m}\|f-\widehat f\|<\varepsilon$ with probability at least $1-\delta$. The useful property is density of the support, rather than continuity of the sampling law: every useful continuous-parameter feature can be approximated arbitrarily closely by a feature whose parameters lie in the discrete support. A practical sampler is multiresolution dyadic quantization, where each coordinate is $q=2^{-r}k$, with integer $k$ and randomly selected precision level $r$. The union over all $r$ is dense, although any individual finite grid is not; therefore the theorem motivates the construction but finite-width performance must be measured empirically.

Implementation notes

(1) Integration point: replace the first dense layer of a small MLP or create a frozen random-feature model. Let $X\in\mathbb{R}^{N\times d}$ be inputs, sample a hidden matrix $W\in\mathbb{R}^{m\times d}$ and bias vector $b\in\mathbb{R}^m$, compute $Z=\phi(XW^\top+b)$, and train only an output matrix $A$. Use ridge regression or gradient descent on $\|ZA-Y\|_F^2$. (2) Pseudocode: for every hidden feature $j$, sample $r_j\sim p_r$, sample integer coordinates $k_{ji}$ and $\ell_j$, set $W_{ji}=2^{-r_j}k_{ji}$ and $b_j=2^{-r_j}\ell_j$, compute $Z=\phi(XW^\top+b)$, and solve $A=(Z^\top Z+\lambda I)^{-1}Z^\top Y$, or optimize the same objective with SGD. Store $k$ and $r$ instead of float weights and reconstruct values during the matrix multiplication. (3) The mathematical constraint to preserve is independent sampling with dense aggregate support; estimate empirically the feature-count requirement, confidence across seeds, and sensitivity to $p_r$, bounds $K_r$, and activation choice. A fixed finite 4-bit grid is an ablation, not a direct dense-support realization. (4) First experiment: use MNIST or CIFAR-10 flattened inputs and compare a Gaussian random-feature baseline, a fixed 4-bit random-feature layer, a fixed 8-bit layer, and the multiresolution dyadic sampler at widths $m\in\{256,512,1024,2048,4096\}$. Repeat each setting over 10 seeds and measure test accuracy, ridge-training time, feature-generation throughput, and hidden-parameter memory. The target signal is accuracy within one percentage point of Gaussian features at equal width, with at least 2x lower hidden-weight storage or faster feature generation using integer/low-bit arithmetic.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.