Rough Multiplicative Random Features

(1) Integrate this at the positional-embedding input of a small Transformer or MLP. For each feature channel j, generate a complex feature \phi_j(n), then concatenate Re\,\phi_j(n) and Im\,\phi_j(n) to the token embedding. Use g_j(x)=\cos(2\pi x), or a short trigonometric polynomial, and choose \alpha_j=(\sqrt{q_j}\bmod 1) for distinct square-free integers q_j. Define the rough position set \mathcal I_z=\{n\le N:P^-(n)>z\}; for positions outside it, either use zero features or a learned fallback embedding. (2) Pseudocode: sample or deterministically hash unit phases u[j,p]; sieve positions up to N to obtain their smallest prime factors; initialize f[j,1]=1; for each n>1 choose any prime divisor p and set f[j,n]=f[j,n/p]u[j,p]; compute y[j,n]=f[j,n]g_j(\alpha_jn); divide by \sqrt{\widehat v_j+\epsilon}; concatenate real and imaginary parts. Because the recursion is valid for complete multiplicativity, factorization costs can be reduced to O(N) preprocessing with a smallest-prime-factor sieve. (3) The paper supplies the multiplicative construction and rough-number Gaussianization principle. Estimate finite-length covariance, fourth-moment excess kurtosis, and variance empirically; do not assume exact independence or finite-length Gaussianity. Compare unrestricted positions with several z values, such as z=1, 3, 7, and a slowly increasing z proportional to \log\log N. (4) First cheap experiment: train a 2-layer Transformer on character-level WikiText-2 or synthetic modular-copy sequences, matching parameter count across learned sinusoidal positions, iid Gaussian random Fourier features, unrestricted multiplicative features, and rough multiplicative features. Record validation loss, long-context extrapolation loss, maximum off-diagonal feature covariance, random-projection kurtosis, positional-feature storage, preprocessing time, and inference latency. Success means matched or improved validation and long-context accuracy with substantially lower stored randomness; reject the method if sieve/factorization overhead dominates or if multiplicative correlations cause optimization instability.