# Directional H-distribution regularizer

- ID: 1661
- Canonical URL: https://synthcore.org/idea/1661/directional-h-distribution-regularizer
- API JSON: https://synthcore.org/api/idea/1661.json
- API Markdown: https://synthcore.org/api/idea/1661.md
- Verification status: unverified
- Source: [arXiv:2608.01216](https://arxiv.org/abs/2608.01216)
- Category: regularization
- Solves: stability, generalization
- ML areas: training, regularization, cnn, transformer
- Math tags: functional-analysis, harmonic-analysis, measure-theory
- Ratings: usefulness 5/10; difficulty 5/10; novelty 7/10

## Idea description

Estimate localized directional Fourier correlations between intermediate activations and their loss residuals, then penalize anisotropic concentration. The method can discourage unstable feature directions and improve robustness without requiring a full microlocal distribution reconstruction.

## Mathematical statement

Assume 1 < p < infinity and p' = p divided by (p minus 1). For sequences u_n converging weakly to zero in local Lp and v_n converging weakly to zero in local Lp', the paper defines an H-distribution mu by the pairing limit: integral of phi_1(x) u_n(x) A_a(phi_2 v_n)(x) over x converges to the pairing of mu with phi_1 phi_2 a. The operator A_a is a Fourier multiplier whose symbol depends only on direction: Fourier(A_a f)(xi) = a(xi divided by |xi|) Fourier(f)(xi), with a a smooth function on the unit sphere. In a minibatch, use feature u_b, residual v_b, spatial window w_r, and directional symbol a_k to estimate C_{r,k} = average over b of the integral w_r u_b A_{a_k}(w_r v_b). Penalize deviations from the directional mean with L_H = lambda times the sum over r and k of the squared difference between C_{r,k} and the mean of C_{r,j} over j. The theorem provides the mathematical motivation for this localized position-direction correlation; the finite-batch loss is an empirical adaptation.

## Key formulas

- $$lim_{n to infinity} integral phi_1(x) u_n(x) A_a(phi_2 v_n)(x) dx = pairing(mu, phi_1 phi_2 a)$$
- $$Fourier(A_a f)(xi) = a(xi divided by |xi|) Fourier(f)(xi)$$
- $$C_{r,k} = (1/B) sum from b=1 to B of integral w_r(x) u_b(x) A_{a_k}(w_r v_b)(x) dx$$
- $$L_H = lambda sum over r and k of (C_{r,k} - (1/K) sum over j of C_{r,j})^2$$

## Implementation notes

Integrate the regularizer on an intermediate CNN feature map or on a ViT patch-token grid reshaped to a two-dimensional lattice. For each sample b, define u_b as the channel mean of the chosen activation and v_b as the corresponding loss residual or gradient with respect to that activation. Detach u_b if the regularizer should affect only the residual pathway; otherwise allow gradients through both tensors. Normalize u and v over each minibatch to remove the zero-frequency mean. Create R Hann or Gaussian spatial windows w_r and K directional masks. For a two-dimensional feature map, construct the FFT frequency grid, compute theta_xi = xi divided by its magnitude, and set a_k(theta_xi) = exp(kappa times the dot product of theta_xi and q_k), where q_k is the unit vector at angle 2 pi k divided by K. Normalize the masks across k and set the zero-frequency mask to zero. Pseudocode: u = normalize(channel_mean(activation); v = normalize(residual); total = 0; for each window r: U = window[r] times u; V = window[r] times v; F = FFT(V); for each direction k: Z = IFFT(F times mask[k]); C[k] = batch_mean(U times Z); total = total + squared(C[k] minus mean(C)); loss = task_loss + lambda times total. The paper supplies the multiplier pairing; FFT resolution, windows, kappa, lambda, and whether to use gradients or residuals are empirical choices. First run CIFAR-10 with ResNet-18, comparing ordinary weight decay with this loss at lambda values 1e-4, 1e-3, and 1e-2, using R = 4 and K = 8. Log clean accuracy, high-frequency corruption accuracy, gradient-norm variance, and directional concentration defined as maximum over k of squared absolute C_{r,k} divided by the sum over j of squared absolute C_{r,j}. A successful result is lower concentration and gradient variance with unchanged clean accuracy, or improved corruption accuracy at equal training FLOPs.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.