Translation-Spectral-Tight Feature Regularization
Implementation & benchmark of arXiv:2608.13142 — Pego theorem for Hilbert space-valued functions on compact groups
Source paper: Pego theorem for Hilbert space-valued functions on compact groups arXiv:2608.13142 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Regularize neural features indexed by a compact transformation group using the three conditions from the vector-valued Pego theorem: nearby group transformations should produce nearby features, high group-Fourier coefficients should have small energy, and feature energy should remain concentrated in a fixed low-dimensional value-space subspace. The third term is important for large or effectively infinite-dimensional feature spaces, because translation and Fourier smoothness alone do not prevent energy from moving into fresh feature directions.
Formulas
Mathematical statement
The paper studies functions f:G o H, where G is a compact group with normalized Haar measure \mu and H is a complex Hilbert space. A unitary representation is a continuous homomorphism \sigma:G o U(H_\sigma), and irreducible representations of compact G are finite-dimensional. For an irreducible representation \sigma of dimension d_\sigma, the vector-valued Fourier coefficient is \widehat f(\sigma)=\int_G f(x)\otimes\sigma(x)^*\,d\mu(x), with Hilbert-Schmidt energy \|\widehat f(\sigma)\|_{\mathrm{HS}}^2. The transferable compactness conditions are bounded L^2 energy, translation equicontinuity \sup_f\| au_yf-f\|_{L^2} o0 as y o e, and uniform Fourier-tail decay. In infinite-dimensional H, the repaired criterion additionally requires uniform tightness: for every \varepsilon>0 there exists a finite-rank orthogonal projection P such that \sup_f\|(I-P)f\|_{L^2}<\varepsilon. We turn these into translation, Fourier-tail, and projection-tail regularizers.
Implementation notes
Integrate the method at the output of an encoder, before its classifier head or residual block. Start with the finite cyclic group C_m, represented by m transformed views g_r x for each input x. Let z_{b,r}=encoder(g_r x_b)\in\mathbb R^h. First compute a translation loss L_trans=(Bm)^{-1}\sum_{b,r}\|z_{b,r+1}-z_{b,r}\|_2^2, with indices modulo m; this discretizes \| au_yf-f\|_{L^2}^2. Next apply an FFT over r: Z_{b,k}=m^{-1}\sum_{r=0}^{m-1}z_{b,r}\exp(-2\pi i kr/m). Select high frequencies K_hi and compute L_fourier=\sum_{b,k\in K_hi}\|Z_{b,k}\|_2^2/(\sum_{b,k}\|Z_{b,k}\|_2^2+\delta), the finite-group analogue of the Fourier tail. Finally maintain an orthonormal rank-q feature basis U. Update an EMA covariance C_t=\beta C_{t-1}+(1-\beta)(Bm)^{-1}\sum_{b,r}z_{b,r}z_{b,r}^{\mathsf T}, and every T steps set U to the top-q eigenvectors of C_t. Compute L_tight=(Bm)^{-1}\sum_{b,r}\|(I-UU^{\mathsf T})z_{b,r}\|_2^2. Optimize L_total=L_task+\lambda_transL_trans+\lambda_fourierL_fourier+\lambda_tightL_tight. The theorem supplies the three structural penalties; m, q, frequency cutoff, EMA coefficient, and weights are empirical hyperparameters. First test CIFAR-10 with C_4 rotations using ResNet-18 and ViT-Tiny. Compare augmentation alone, translation consistency, translation plus Fourier regularization, and all three terms. Measure clean and transformed accuracy, feature drift across transformations, effective covariance rank, activation memory, and training-loss variance. Success is higher worst-case transformed accuracy or lower feature drift at equal clean accuracy, with reduced effective feature rank and no substantial optimization slowdown.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.