# Exact energy-preserving activation subsampling

- ID: 155
- Canonical URL: https://synthcore.org/idea/155/exact-energy-preserving-activation-subsampling
- API JSON: https://synthcore.org/api/idea/155.json
- API Markdown: https://synthcore.org/api/idea/155.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.25058](https://arxiv.org/abs/2608.25058)
- Category: regularization
- Solves: memory, stability, speedup
- ML areas: mlp, transformer, moe-routing, regularization, memory
- Math tags: linear-algebra, harmonic-analysis, combinatorics
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Use the weighted quadrature identity as a training or inference constraint for a compressed activation path: retain only a minimal set of binary evaluations and compute normalization or residual-energy statistics exactly on the modeled Rademacher component. This provides a zero-variance alternative to random activation subsampling for the represented subspace.

## Mathematical statement

For c=(c_0,...,c_{N-1})^T and selected sign vectors a_k=(r_0(Delta_{i_k}),...,r_{N-1}(Delta_{i_k}))^T, the paper requires sum_k lambda_k a_k a_k^T=I_N. Consequently E(c)=sum_k lambda_k(a_k^T c)^2 equals c^T c for every coefficient vector, not merely in expectation. The full dyadic construction uses 2^N cells with uniform weights 2^{-N}; the paper's objective is to set as many weights to zero as possible, resulting in minimum support N or N+1. In a neural module, E(c) can replace a sampled estimate of the squared norm, RMS statistic, residual penalty, or router auxiliary statistic for the binary-feature component.

## Key formulas

- $$\sum_{k=1}^{m}\lambda_k a_k a_k^{\mathsf T}=I_N\quad\Longleftrightarrow\quad \sum_{k=1}^{m}\lambda_k(a_k^{\mathsf T}c)^2=\|c\|_2^2\ \text{for all }c\in\mathbb R^N$$
- $$\int_0^1f^2(x)\,dx=\sum_{k=1}^{m}\lambda_kf(\xi_k)^2,\qquad f(x)=\sum_{l=0}^{N-1}c_lr_l(x)$$
- $$\int_0^1 f^2(x)\,dx=\frac{1}{2^N}\sum_{i=0}^{2^N-1}f(\xi^i)^2,\qquad \xi^i\in\Delta_i$$

## Implementation notes

Apply this to a binary-feature branch inside an MLP, residual block, or router whose logits are generated from sign features. Given coefficients `c = g_theta(x)` with shape [batch,N], precompute selected sign vectors A and nonnegative weights lambda satisfying A diag(lambda) A^T=I. Compute only `z = c @ A`, then the exact modeled energy as `E = ((z*z) * lambda).sum(dim=-1)`. Use E instead of a randomly subsampled squared norm, RMS denominator, residual scaling statistic, or auxiliary load-balancing penalty. For arbitrary activations q, train a small linear projector P and apply the exact estimator to c=Pq; retain a cheap Monte Carlo estimator for the complement q-Pq. During implementation, verify the matrix residual norm of A diag(lambda) A^T-I and test numerical drift in mixed precision. The paper supplies the zero-variance identity; projection quality is empirical. Start with CIFAR-100 or WikiText-103 using a small MLP or transformer, and subsample 16 of 256 binary feature evaluations. Compare uniform random sampling, importance sampling, and the exact design at equal activation memory. Measure norm-estimator variance, gradient variance, loss smoothness, wall-clock time, and validation accuracy. The desired result is lower statistic and gradient variance, smoother training, and equal or better accuracy at the same activation budget.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built an exact Hadamard-sign energy estimator for a 16-dimensional modeled coefficient vector, plus a matched random-sign baseline and toy normalized MLP regression experiment. The matrix residual was exactly 0 and energy error was 6.1e-16, while random sampling had 0.323 relative RMSE and nonzero gradient variance; at 220 steps, exact normalization reached 0.00140 loss versus 0.0326 for random sampling at essentially equal runtime. This is a clear signal for the represented subspace, though not evidence of broad neural-network gains.

### Mechanism check

- Verdict: Built an exact Hadamard-sign energy estimator for a 16-dimensional modeled coefficient vector, plus a matched random-sign baseline and toy normalized MLP regression experiment. The matrix residual was exactly 0 and energy error was 6.1e-16, while random sampling had 0.323 relative RMSE and nonzero gradient variance; at 220 steps, exact normalization reached 0.00140 loss versus 0.0326 for random sampling at essentially equal runtime. This is a clear signal for the represented subspace, though not evidence of broad neural-network gains.
- Confidence: 9/10
- Limitations: Only a tiny synthetic regression task and N=16 full-support Hadamard design were tested; the proposed 16-of-256 setting, learned projection/complement, CIFAR/WikiText/transformer or MoE workloads, wall-clock memory savings, importance sampling, and larger-scale generalization were not tested. Randomness is in the sign design rather than data minibatching.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.64985
- Paired wins: 4/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/1190/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/1190/bench_report.json)
- [experiment.py](https://synthcore.org/code/1190/experiment.py)
- [report.md](https://synthcore.org/code/1190/report.md)
- [report_bench_2026-09-03T030916.md](https://synthcore.org/code/1190/report_bench_2026-09-03T030916.md)
- [results.json](https://synthcore.org/code/1190/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1190)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.