# Private spectral whitening front-end

- ID: 145
- Canonical URL: https://synthcore.org/idea/145/private-spectral-whitening-front-end
- API JSON: https://synthcore.org/api/idea/145.json
- API Markdown: https://synthcore.org/api/idea/145.md
- Verification status: beats_baseline
- Source: [arXiv:2608.24847](https://arxiv.org/abs/2608.24847)
- Category: architecture
- Solves: stability, sample-efficiency, accuracy
- ML areas: transformer, ssm, attention, federated
- Math tags: harmonic-analysis, statistics, probability, spectral-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Estimate the temporal spectrum of each sequence channel using a locally private procedure, then apply a regularized inverse-square-root spectral filter before the sequence enters attention or an SSM. The filter removes predictable low-frequency or narrow-band redundancy while avoiding unstable amplification at frequencies where the private estimate is small.

## Mathematical statement

The paper defines the autocovariance of a centered stationary process X_t by gamma(h) = Cov(X_0,X_h), where h is an integer lag, and its spectral density by f(lambda) = (1/(2 pi)) sum_h gamma(h) exp(-i h lambda), where lambda belongs to [-pi, pi]. For a real-valued stationary Gaussian process, f is nonnegative, integrable, and even. We use the corresponding filter W(lambda) = (max(f_hat(lambda), epsilon))^(-1/2), where f_hat is a privatized spectral estimate and epsilon > 0 is a numerical floor. If X has spectrum f, the filtered signal Y = W * X has approximate spectrum f_Y(lambda) = |W(lambda)|^2 f(lambda), which is close to one when f_hat is accurate, thereby flattening temporal correlations. The paper's central privacy result gives an effective sample size N_eff asymptotically proportional to N alpha^4 in the high-privacy regime for dependent Gaussian observations, where N is the number of observations and alpha is the local differential privacy parameter. This scaling determines how strongly the spectrum should be smoothed.

## Key formulas

- $$\gamma(h)=\operatorname{Cov}(X_{0},X_{h}),\qquad f(\lambda)=\frac{1}{2\pi}\sum_{h\in\mathbb{Z}}\gamma(h)e^{-ih\lambda},\quad \lambda\in[-\pi,\pi].$$
- $$N_{\mathrm{eff}}\asymp N\alpha^{4}\quad\text{in the high-privacy regime for dependent Gaussian observations}.$$
- $$W(\lambda)=\left(\max\{\widehat f(\lambda),\varepsilon\}\right)^{-1/2},\qquad \widetilde X(\lambda)=W(\lambda)X(\lambda).$$
- $$\widehat X_t=\operatorname{IFFT}\left(W(\lambda_k)\operatorname{FFT}(X)_k\right)_t.$$

## Implementation notes

Integrate this module immediately before the first temporal attention block or SSM, operating independently on each scalar feature channel or on small groups of channels. For a minibatch of length-T sequences, first clip each channel to a fixed range [-B,B]. Compute a periodogram I_b(k) = T^(-1)|FFT(x_b)_k|^2 and aggregate it across examples. For a locally private MVP, privatize clipped cosine and sine autocovariance features with Gaussian local noise or randomized response; after reading the full paper, replace this provisional mechanism with its problem-specific optimal estimator. Form f_hat_k, symmetrize it using (f_hat_k + f_hat_(T-k))/2, project negative values to zero, and smooth across neighboring frequencies. Set N_eff = N alpha^4 and use stronger smoothing when N_eff is small. Compute W_k = (max(f_hat_k, epsilon))^(-1/2), multiply each FFT channel by W_k, and apply an inverse FFT. Keep the filter fixed within an epoch or update it only every several hundred batches to avoid noisy gradients. The paper supplies the Fourier identity and privacy scaling; the engineer estimates f_hat, B, the smoothing bandwidth, and epsilon empirically. Begin with a 2-layer Transformer on synthetic AR(1), long-memory, and white-noise sequences, then test WikiText-2 or a comparable sequence dataset. Compare raw inputs, nonprivate whitening, and private whitening at equal privacy budget. Success means lower validation loss, reduced gradient-norm variance, or matching baseline perplexity with fewer private sequences.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: helps
- Verdict: Built a NumPy MVP for clipped noisy-autocovariance spectral estimation and regularized FFT whitening. The Fourier identity was verified to 1.1e-16 error, exact filtering flattened the spectrum with CV 0.089, and private whitening reduced held-out AR(1) lag-autocorrelation energy from 0.190 to 0.0369 at alpha=1. The effect is real but weaker than nonprivate whitening (0.00150), and stronger privacy degraded spectral MSE substantially.

### Mechanism check

- Verdict: Built a NumPy MVP for clipped noisy-autocovariance spectral estimation and regularized FFT whitening. The Fourier identity was verified to 1.1e-16 error, exact filtering flattened the spectrum with CV 0.089, and private whitening reduced held-out AR(1) lag-autocorrelation energy from 0.190 to 0.0369 at alpha=1. The effect is real but weaker than nonprivate whitening (0.00150), and stronger privacy degraded spectral MSE substantially.
- Confidence: 8/10
- Limitations: Only synthetic AR(1) decorrelation was tested; no Transformer or SSM validation loss, gradient variance, language data, long-memory or white-noise benchmarks, GPU path, formal LDP accounting, or rigorous alpha^4 rate fit was included. The Gaussian mechanism is the provisional clipped-feature MVP rather than the paper's optimal estimator.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_private_whitening.py](https://synthcore.org/code/1187/bench_private_whitening.py)
- [bench_report.json](https://synthcore.org/code/1187/bench_report.json)
- [private_spectral_whitening.py](https://synthcore.org/code/1187/private_spectral_whitening.py)
- [report.md](https://synthcore.org/code/1187/report.md)
- [report_bench_2026-09-03T213917.md](https://synthcore.org/code/1187/report_bench_2026-09-03T213917.md)
- [results.json](https://synthcore.org/code/1187/results.json)
- [stage2_bench.py](https://synthcore.org/code/1187/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1187)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.