# Rough Multiplicative Random Features

- ID: 1622
- Canonical URL: https://synthcore.org/idea/1622/rough-multiplicative-random-features
- API JSON: https://synthcore.org/api/idea/1622.json
- API Markdown: https://synthcore.org/api/idea/1622.md
- Verification status: unverified
- Source: [arXiv:2608.00184](https://arxiv.org/abs/2608.00184)
- Category: architecture
- Solves: memory, generalization
- ML areas: embedding, transformer, initialization, regularization
- Math tags: probability, stochastic-processes, number-theory, harmonic-analysis
- Ratings: usefulness 4/10; difficulty 5/10; novelty 9/10

## Idea description

Replace explicitly stored independent random positional features with deterministic multiplicative phase features generated from a small table of random phases indexed by primes. Restrict positions to integers whose prime factors exceed a slowly growing threshold, because the paper's central-limit result says periodic weighted sums over these rough integers recover Gaussian behavior despite strong multiplicative dependence.

## Mathematical statement

A Steinhaus random multiplicative function assigns independent random variables f(p), uniformly distributed on the complex unit circle, to each prime p and extends them completely multiplicatively by f(n):=\prod_{p^{k}\parallel n}f(p)^k, where p^k\parallel n means p^k divides n but p^{k+1} does not. For a 1-periodic bounded-variation function g and irrational \alpha, the paper studies S_N=\sum_{n\le N}f(n)g(\alpha n). Its transferable result is that restricting the sum to \mathcal S_z=\{n:P^-(n)>z\}, where P^-(n) is the smallest prime factor of n and z tends to infinity arbitrarily slowly, yields a variance-normalized central limit theorem. The fourth-moment calculation identifies the cancellation mechanism: for Q(    heta)=\sum_{|\ell|\le L}c_\ell e(\ell    heta), with e(x)=\exp(2\pi i x), non-diagonal terms contain phases e[    heta\psi(\mathbf n,\boldsymbol\ell)] where \psi=n_1\ell_1+n_2\ell_2-n_3\ell_3-n_4\ell_4, while multiplicative dependencies satisfy n_1n_2=n_3n_4. The rough-prime restriction suppresses exceptional collisions. In the neural feature construction, j indexes channels, p indexes primes, n is position, \alpha_j is an irrational frequency, and \widehat v_j is an empirical feature variance.

## Key formulas

- $$f(n):=\prod_{p^{k}\parallel n}f(p)^k,\qquad f(p)\stackrel{\mathrm{iid}}{\sim}\mathrm{Unif}\{z\in\mathbb C:|z|=1\}.$$
- $$S_N(\alpha,g;z):=\sum_{\substack{n\le N\\P^-(n)>z}}f(n)g(\alpha n),\qquad \frac{S_N(\alpha,g;z)}{\sqrt{\mathbb E|S_N(\alpha,g;z)|^2}}\Rightarrow\mathcal{CN}(0,1)\quad\text{as }N\to\infty,\ z\to\infty\text{ arbitrarily slowly}.$$
- $$Q(\theta)=\sum_{|\ell|\le L}c_\ell e(\ell\theta),\qquad \psi(\mathbf n,\boldsymbol\ell)=n_1\ell_1+n_2\ell_2-n_3\ell_3-n_4\ell_4.$$
- $$\phi_j(n):=\frac{f_j(n)g_j(\alpha_j n)}{\sqrt{\widehat v_j}},\qquad \widehat v_j=\frac{1}{|\mathcal I|}\sum_{n\in\mathcal I}|f_j(n)g_j(\alpha_j n)|^2.$$

## Implementation notes

(1) Integrate this at the positional-embedding input of a small Transformer or MLP. For each feature channel j, generate a complex feature \phi_j(n), then concatenate Re\,\phi_j(n) and Im\,\phi_j(n) to the token embedding. Use g_j(x)=\cos(2\pi x), or a short trigonometric polynomial, and choose \alpha_j=(\sqrt{q_j}\bmod 1) for distinct square-free integers q_j. Define the rough position set \mathcal I_z=\{n\le N:P^-(n)>z\}; for positions outside it, either use zero features or a learned fallback embedding. (2) Pseudocode: sample or deterministically hash unit phases u[j,p]; sieve positions up to N to obtain their smallest prime factors; initialize f[j,1]=1; for each n>1 choose any prime divisor p and set f[j,n]=f[j,n/p]u[j,p]; compute y[j,n]=f[j,n]g_j(\alpha_jn); divide by \sqrt{\widehat v_j+\epsilon}; concatenate real and imaginary parts. Because the recursion is valid for complete multiplicativity, factorization costs can be reduced to O(N) preprocessing with a smallest-prime-factor sieve. (3) The paper supplies the multiplicative construction and rough-number Gaussianization principle. Estimate finite-length covariance, fourth-moment excess kurtosis, and variance empirically; do not assume exact independence or finite-length Gaussianity. Compare unrestricted positions with several z values, such as z=1, 3, 7, and a slowly increasing z proportional to \log\log N. (4) First cheap experiment: train a 2-layer Transformer on character-level WikiText-2 or synthetic modular-copy sequences, matching parameter count across learned sinusoidal positions, iid Gaussian random Fourier features, unrestricted multiplicative features, and rough multiplicative features. Record validation loss, long-context extrapolation loss, maximum off-diagonal feature covariance, random-projection kurtosis, positional-feature storage, preprocessing time, and inference latency. Success means matched or improved validation and long-context accuracy with substantially lower stored randomness; reject the method if sieve/factorization overhead dominates or if multiplicative correlations cause optimization instability.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.