Fourier moment-capped cyclic layers
Implementation & benchmark of arXiv:2607.13666 — Spectral and Additive Combinatorial Methods for Cycles and Absorbing Sets in Lifted-Product Quantum LDPC Codes
Source paper: Spectral and Additive Combinatorial Methods for Cycles and Absorbing Sets in Lifted-Product Quantum LDPC Codes arXiv:2607.13666 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace expensive global spectral diagnostics of a cyclic or block-circulant neural layer by exact small Fourier-block calculations. Add a scale-normalized fourth-moment penalty, or directly cap the largest eigenvalue of each frequency block, to suppress frequency-specific amplification and reduce unstable training in long cyclic convolutions and structured attention.
Formulas
Mathematical statement
The paper's Theorem 4.1 states that for a lifted-product block, the large Gram matrix has Fourier blocks M_j=A(omega^j)A(omega^j)^*+|b(omega^j)|^2 I_{r_1}, where j is a cyclic frequency in {0,...,ell-1}, omega=exp(2 pi i/ell), A(omega^j) is the Fourier evaluation of the first block, b(omega^j) is the scalar Fourier evaluation of the second block, I_{r_1} is the r_1 by r_1 identity, and * denotes conjugate transpose. Each M_j is positive semidefinite. The exact trace identity is tr((H_X H_X^T)^k)=sum_j tr(M_j^k)=sum_j sum_i lambda_i(M_j)^k, where k is a positive integer and lambda_i(M_j) are nonnegative eigenvalues. For a neural layer, let W_j be its Fourier-domain channel-mixing matrix and define M_j=W_j W_j^*+epsilon I, with epsilon>0. Use the adapted scale-normalized moment R_4=(ell r_1)^(-1) sum_j tr((M_j/bar_mu)^4), where bar_mu=(ell r_1)^(-1) sum_j tr(M_j) is detached from gradients. This penalizes heavy spectral tails without simply driving all weights to zero.
Implementation notes
Integrate this into a cyclic 1D convolution, block-circulant linear layer, or attention module whose sequence dimension has length ell. The layer should expose its channel-mixing kernels as tensors with shape [out_channels, in_channels, ell], or as a shorter kernel embedded into an ell-point circular convolution. First compute the unitary FFT of the kernel along the cyclic dimension, producing W_j for every frequency j. Form M_j=W_j W_j^*+epsilon I without materializing the ell-by-ell convolution matrix. For the soft-regularization variant, compute bar_mu=(ell r_1)^(-1) sum_j tr(M_j) using stop-gradient, compute R_4=(ell r_1)^(-1) sum_j tr((M_j/bar_mu)^4) by matrix multiplications, and add lambda R_4 to the task loss. For a hard stability variant, estimate the largest eigenvalue of each M_j with two or three power iterations and apply the supplied rescaling s_j to the Fourier block, followed by inverse FFT before the ordinary forward pass. The exact mathematics supplies the block decomposition and trace moment; lambda, tau, and epsilon are empirical hyperparameters. Start with a small circular CNN on CIFAR-10 or a tiny structured Transformer on WikiText-2, comparing ordinary weight decay and spectral normalization at equal FLOPs. Measure gradient spikes, per-frequency gain, validation loss, and FFT overhead. Success means fewer instability events and lower validation loss or faster loss descent with less than 10% overhead.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.