Quadratic Gaussianization for Sign Layers
Source paper: A Sharp Small-Coefficient Variant of Khintchine's Inequality and the Sharp $π/2$ Theorem arXiv:2608.29703 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Add coefficient-spreading and moment-calibration mechanisms to binary or sign-noised linear layers. For each output neuron, normalize its real-valued weights and penalize large normalized coordinates, so its signed preactivation obeys the paper's quadratic Gaussian approximation rather than the weaker linear bound. This should make activation scales more predictable and reduce training instability caused by highly concentrated binary projections.
Formulas
Mathematical statement
Let X_1,...,X_n be independent Rademacher variables with P(X_i=1)=P(X_i=-1)=1/2, let w=(w_1,...,w_n) satisfy ||w||_2^2=sum_i w_i^2=1, and define W=sum_i w_i X_i. The classical estimate in the extracted text is |E|W|-sqrt(2/pi)| <= (1-sqrt(2/pi)) beta, where beta=||w||_infinity=max_i |w_i|. The paper's main refinement states that for sufficiently small beta, uniformly over all admissible w, |E|W|-sqrt(2/pi)| <= C beta^2, for a universal constant C; the extremal construction shows that the quadratic order is sharp. Since E|Z|=sqrt(2/pi) for Z~N(0,1), spreading the normalized coefficients makes the absolute activation moment quadratically close to the Gaussian target. In a neural layer with sign inputs, each row's preactivation is exactly such a W. We use beta(w)=||w||_infinity/(||w||_2+epsilon) as the coefficient-concentration statistic and penalize beta(w)^2. An optional empirical moment term targets the Gaussian absolute first moment; for minibatch preactivations a_bj, define m_j=(B^{-1} sum_b |a_bj|)/(sqrt(B^{-1} sum_b a_bj^2)+epsilon), then penalize (m_j-sqrt(2/pi))^2.
Implementation notes
Integrate this into a binary linear layer, sign-activation MLP, or real-valued layer receiving sign-noised inputs. For each trainable output row v_j in R^n, compute w_j=v_j/(||v_j||_2+epsilon) before the matrix multiplication. For minibatch input x, set s=sign(x) using the straight-through estimator, or use independently sampled Rademacher noise s_i in a controlled ablation, and compute a=s W^T. Then calculate beta_j=||v_j||_infinity/(||v_j||_2+epsilon). The coefficient term is lambda_beta times the mean of beta_j squared. For the optional moment term, calculate m_j=(mean_b |a_bj|)/(sqrt(mean_b a_bj^2)+epsilon), and add lambda_m times (m_j-sqrt(2/pi)) squared. Compact pseudocode is: v <- parameters; w <- v/(row_l2(v)+eps); s <- STE_sign(x) or random_Rademacher_like(x); a <- s @ w^T; beta <- row_linf(v)/(row_l2(v)+eps); m <- mean_batch(abs(a))/(sqrt(mean_batch(a^2))+eps); loss <- task_loss + lambda_beta*mean(beta^2) + lambda_m*mean((m-sqrt(2/pi))^2); backpropagate. The paper-derived quantities are the target sqrt(2/pi) and the predicted quadratic dependence on beta; the unknown constant C should be estimated by regressing measured moment error against beta squared, not hard-coded. The first cheap experiment is CIFAR-10 with a three-layer sign-MLP or small binary ResNet. Compare baseline weight normalization, beta regularization alone, and the full QG loss at matched parameter count and binary FLOPs. Record validation accuracy, training-loss variance, activation-scale drift, fraction of saturated sign units, and the plot of |m_j-sqrt(2/pi)| versus beta_j and beta_j squared. The method is supported if activation drift and optimization spikes decrease, and if the empirical error is more linearly correlated with beta squared than beta at small beta, without sacrificing accuracy.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.