Degree-Aware Tensor Concentration Clipper
Implementation & benchmark of arXiv:2608.19832 — Sharp Convex Concentration for Symmetric Random Tensors with Subgaussian Coordinates
Source paper: Sharp Convex Concentration for Symmetric Random Tensors with Subgaussian Coordinates arXiv:2608.19832 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Add a calibrated robustification rule after a symmetric polynomial feature map z(x)=vec(x^{\otimes d}). For a convex Lipschitz head or loss applied to z(x), compute a high-probability deviation radius from the paper's concentration rate and clip only examples beyond that radius. This explicitly accounts for the large radial fluctuations created by reusing the same vector in every tensor slot.
Formulas
Mathematical statement
Let X=(X_1,...,X_n) have independent mean-zero variance-one coordinates with subgaussian norm ||X_i||_{\psi_2}<=K, let H_d=(R^n)^{\otimes_2 d}, and let f:H_d->R be convex and L-Lipschitz in Euclidean norm. For Z=X^{\otimes d}, the paper proves, for 0<=t<=c_K L n^{d/2}, P(|f(Z)-E f(Z)|>t)<=C exp[-c_K I_{n,d}(t/(L n^{(d-1)/2}))]. Here c_K and C depend only on K, and I_{n,d}(s)=min{s^2/d^2, s^2/[d log(e+nd/s^2)]}. The first branch is caused by radial variation because ||X^{\otimes d}||_2=||X||_2^d; the second controls directional changes at nearly fixed norm. For target failure probability delta, define u=log(C/delta)/c_K and choose the smallest s_delta satisfying I_{n,d}(s_delta)>=u. The clipping radius is t_delta=L n^{(d-1)/2}s_delta. For a linear head followed by a 1-Lipschitz scalar loss, use L approximately equal to the spectral norm ||W||_2.
Implementation notes
(1) Integration point: use a degree-2 or degree-3 symmetric tensorized MLP. Given standardized features h in R^{B x n}, form z_b=vec(h_b^{\otimes d}), pass z_b through a linear or shallow convex head, and compute a per-example loss ell_b. Apply the rule to each loss or to its per-example gradient contribution before batch averaging. For a linear head and a 1-Lipschitz scalar loss, estimate L by power iteration on W and use L=||W||_2.
(2) Pseudocode: standardize h; compute z_b=h_b^{\otimes d}; evaluate ell_b; update the running center bar_ell using an exponential moving average; set u=log(C/delta)/c_K; solve by bisection for the smallest positive s satisfying min(s^2/d^2, s^2/(d*log(e+n*d/s^2)))>=u; set t_delta=L*n^((d-1)/2)*s; replace each loss deviation by sign(ell_b-bar_ell)*min(abs(ell_b-bar_ell),t_delta), or multiply its gradient by min(1,t_delta/abs(ell_b-bar_ell)).
(3) The paper supplies the tensor scaling and the two-branch rate. Estimate L with power iteration, estimate K from coordinatewise empirical subgaussian tails, and estimate bar_ell online. Whiten inputs before applying the rule; correlated coordinates violate the theorem's main assumption and should be tested separately. Keep t_delta within the stated theorem range when possible.
(4) First experiment: train degree-2 and degree-3 tensorized MLPs on n=32 synthetic Gaussian regression and classification, followed by whitened MNIST patches. Compare no clipping, global gradient-norm clipping, percentile loss clipping, and this rule at matched clipping frequency. Record gradient norm spikes, loss variance, fraction of clipped examples, steps to target training loss, and final test error at equal FLOPs. Success is fewer extreme gradient events and lower or equal test error without slower loss descent; failure on correlated unwhitened inputs would identify the assumption boundary.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.