Uniform Lp Activation Sketch

(1) Integration point: insert the sketch after a wide M-dimensional activation vector h, preferably after a 1x1 convolution, MLP expansion, or token-wise feed-forward block. First collect a calibration matrix H\in\mathbb R^{B imes M} from several batches, compute a rank-d basis U\in\mathbb R^{M imes d} by randomized SVD, and represent h\approx Ua. The deployable layer stores only selected coordinate indices i_j and positive weights w_j; its output is the N-vector (w_j^{1/p}h_{i_j})_j. A downstream linear layer must be folded through the corresponding weighted coordinate map or retrained after insertion. (2) Pseudocode: compute U; initialize candidate coordinates using leverage scores q_i=\|u_i\|_2^2/\sum_k\|u_k\|_2^2; repeatedly add or swap coordinates; fit nonnegative weights by minimizing the maximum sampled-sphere error E(a)=|\sum_j w_j|u_{i_j}^Ta|^p/(\sum_i|u_i^Ta|^p)-1|; approximate the maximum with K random sphere points plus adversarial gradient ascent on a; stop when max E(a)\leq\epsilon. Use projected Adam or nonnegative least squares for w. (3) Computed from the mathematics: the desired N scaling N\propto\epsilon^{-2(d-1)/(d+2p)}, the uniform sphere objective, and the weighted \ell_p embedding. Estimated empirically: d from explained variance, the sphere maximum from adversarial samples, and the final distortion on held-out activation combinations. For p=2k, additionally test exact low-order moment matching because |t|^p=t^{2k} is a polynomial; for non-even p, use denser sphere/adversarial sampling to cover the approximation remainder. (4) First experiment: compress the expansion dimension of a small Transformer FFN or a CNN channel block on CIFAR-10 or WikiText-2, with d\in\{4,8,16\}, p\in\{1,2,3\}, and N/M\in\{0.25,0.5\}. Compare magnitude pruning, random coordinate sampling, leverage-score sampling, and the uniform Lp sketch. Measure worst-case relative p-norm distortion on held-out linear combinations, downstream validation accuracy or perplexity after brief fine-tuning, wall-clock inference time, and activation memory. Success is lower uniform distortion and at least 1.5x lower activation bandwidth at matched accuracy; the paper-inspired scaling predicts that increasing N should reduce distortion according to the stated \epsilon-exponent rather than the usual Monte Carlo N^{-1/2} law.