arXiv:2607.15598 2026 Regularization 1 ideas extracted · analyzed Aug 30, 2026
The paper defines a scale-invariant Fourier ratio for a matrix and characterizes extreme spectral concentration through Fourier structure. This offers a transferable complexity measure for learned interaction kernels whose indices have a meaningful spatial or cyclic ordering, especially relative-position biases in vision transformers. The most practical adaptation is to penalize the Fourier ratio during training and then retain only the dominant frequency coefficients at inference, producing a compact bias representation. The approach is falsifiable through accuracy, retained-frequency count, memory, and latency measurements against unrestricted and sparsity-regularized baselines.
Replace an unconstrained relative-position attention-bias table by a learnable two-dimensional Fourier representation, or regularize the bias toward a small Fourier ratio. The ratio favors coefficient concentration without depending on the absolute scale of the bias, allowing the trained bias to be reconstructed from a small number of dominant frequencies.