△ Mechanism confirmed, baseline not beaten 2026

Exact energy-preserving activation subsampling

Implementation & benchmark of arXiv:2608.25058 — On exact discretization of the $L_2$-norm in the space spanned by the first $N$ Rademacher functions

Usefulness6/10

Difficulty4/10

Novelty6/10

Regularization Memory Stability Speedup Mlp Transformer Moe routing Regularization Memory Linear algebra Harmonic analysis Combinatorics

Source paper: On exact discretization of the $L_2$-norm in the space spanned by the first $N$ Rademacher functions arXiv:2608.25058 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use the weighted quadrature identity as a training or inference constraint for a compressed activation path: retain only a minimal set of binary evaluations and compute normalization or residual-energy statistics exactly on the modeled Rademacher component. This provides a zero-variance alternative to random activation subsampling for the represented subspace.

Formulas

$$\sum_{k=1}^{m}\lambda_k a_k a_k^{\mathsf T}=I_N\quad\Longleftrightarrow\quad \sum_{k=1}^{m}\lambda_k(a_k^{\mathsf T}c)^2=\|c\|_2^2\ \text{for all }c\in\mathbb R^N$$

$$\int_0^1f^2(x)\,dx=\sum_{k=1}^{m}\lambda_kf(\xi_k)^2,\qquad f(x)=\sum_{l=0}^{N-1}c_lr_l(x)$$

$$\int_0^1 f^2(x)\,dx=\frac{1}{2^N}\sum_{i=0}^{2^N-1}f(\xi^i)^2,\qquad \xi^i\in\Delta_i$$

Mathematical statement

For c=(c_0,...,c_{N-1})^T and selected sign vectors a_k=(r_0(Delta_{i_k}),...,r_{N-1}(Delta_{i_k}))^T, the paper requires sum_k lambda_k a_k a_k^T=I_N. Consequently E(c)=sum_k lambda_k(a_k^T c)^2 equals c^T c for every coefficient vector, not merely in expectation. The full dyadic construction uses 2^N cells with uniform weights 2^{-N}; the paper's objective is to set as many weights to zero as possible, resulting in minimum support N or N+1. In a neural module, E(c) can replace a sampled estimate of the squared norm, RMS statistic, residual penalty, or router auxiliary statistic for the binary-feature component.

Implementation notes

Apply this to a binary-feature branch inside an MLP, residual block, or router whose logits are generated from sign features. Given coefficients c = g_theta(x) with shape [batch,N], precompute selected sign vectors A and nonnegative weights lambda satisfying A diag(lambda) A^T=I. Compute only z = c @ A, then the exact modeled energy as E = ((z*z) * lambda).sum(dim=-1). Use E instead of a randomly subsampled squared norm, RMS denominator, residual scaling statistic, or auxiliary load-balancing penalty. For arbitrary activations q, train a small linear projector P and apply the exact estimator to c=Pq; retain a cheap Monte Carlo estimator for the complement q-Pq. During implementation, verify the matrix residual norm of A diag(lambda) A^T-I and test numerical drift in mixed precision. The paper supplies the zero-variance identity; projection quality is empirical. Start with CIFAR-100 or WikiText-103 using a small MLP or transformer, and subsample 16 of 256 binary feature evaluations. Compare uniform random sampling, importance sampling, and the exact design at equal activation memory. Measure norm-estimator variance, gradient variance, loss smoothness, wall-clock time, and validation accuracy. The desired result is lower statistic and gradient variance, smoother training, and equal or better accuracy at the same activation budget.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built an exact Hadamard-sign energy estimator for a 16-dimensional modeled coefficient vector, plus a matched random-sign baseline and toy normalized MLP regression experiment. The matrix residual was exactly 0 and energy error was 6.1e-16, while random sampling had 0.323 relative RMSE and nonzero gradient variance; at 220 steps, exact normalization reached 0.00140 loss versus 0.0326 for random sampling at essentially equal runtime. This is a clear signal for the represented subspace, though not evidence of broad neural-network gains.

Agent confidence: 9/10

Baseline: Random 16-sign estimator: relative energy RMSE 0.3232, mean relative estimator std 0.3223, mean gradient element variance 4.06e-7, final 20-step MLP loss 0.03258, runtime 1.0097 s
Idea: 16-column Hadamard exact estimator: matrix Frobenius residual 0, maximum relative energy error 6.06e-16, mean gradient element variance 3.68e-19, final 20-step MLP loss 0.001396, runtime 1.0264 s; float16 input with float32 accumulation max relative error 7.87e-4

Limitations: Only a tiny synthetic regression task and N=16 full-support Hadamard design were tested; the proposed 16-of-256 setting, learned projection/complement, CIFAR/WikiText/transformer or MoE workloads, wall-clock memory savings, importance sampling, and larger-scale generalization were not tested. Randomness is in the sign design rather than data minibatching.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.25058) Sep 3, 2026

No significant difference vs. the tuned baseline (+0.3%, p=0.6499, wins 4 / 8 paired seeds).

Benchmark: Vision (image classification)
Model: cnn_small
Paired seeds: 8
Baseline mean: 0.785
Idea mean: 0.7872
Effect (Δ): 0.0022 (+0.3%; negative = idea better)
Wins: 4 / 8 paired seeds
p-value: 0.6499 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched vision track was tested, using 1000 training and 400 test CIFAR-10 examples, 8 epochs, and N=16 evaluations rather than the proposed 16-of-256 projection/complement setting. Wall-clock activation-memory savings, importance sampling, learned projections, mixed precision, and larger datasets were not benchmarked.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 3, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented exact Hadamard energy-preserving activation normalization in a matched CNN system and ran the full 8-seed vision benchmark with a shared learning-rate union and baseline sweep. The mechanism prediction was confirmed on trained models: exact relative energy RMSE was 5.17e-8 versus random-estimator relative standard deviation 0.3648. Test error was slightly worse for the idea (0.78718749 vs 0.78499998), with paired delta +0.00218751 and permutation p=0.64985, so there was no benchmark win.

Baseline: vision/cnn_small, best lr=0.001, test error mean 0.78499998; per-seed [0.78750, 0.75000, 0.83500, 0.77250, 0.82750, 0.81750, 0.72500, 0.76500]
Idea: Exact Hadamard normalization, best lr=0.001, test error mean 0.78718749; per-seed [0.79000, 0.77000, 0.84000, 0.76250, 0.82500, 0.80750, 0.74750, 0.75500]

Benchmark result