✓✓ Beats tuned baseline 2026

Private spectral whitening front-end

Implementation & benchmark of arXiv:2608.24847 — On the privacy cost for dependent Gaussian data: spectral density estimation under local differential privacy

Usefulness6/10

Difficulty5/10

Novelty7/10

Architecture Stability Sample efficiency Accuracy Transformer Ssm Attention Federated Harmonic analysis Statistics Probability Spectral theory

Source paper: On the privacy cost for dependent Gaussian data: spectral density estimation under local differential privacy arXiv:2608.24847 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Estimate the temporal spectrum of each sequence channel using a locally private procedure, then apply a regularized inverse-square-root spectral filter before the sequence enters attention or an SSM. The filter removes predictable low-frequency or narrow-band redundancy while avoiding unstable amplification at frequencies where the private estimate is small.

Formulas

$$\gamma(h)=\operatorname{Cov}(X_{0},X_{h}),\qquad f(\lambda)=\frac{1}{2\pi}\sum_{h\in\mathbb{Z}}\gamma(h)e^{-ih\lambda},\quad \lambda\in[-\pi,\pi].$$

$$N_{\mathrm{eff}}\asymp N\alpha^{4}\quad\text{in the high-privacy regime for dependent Gaussian observations}.$$

$$W(\lambda)=\left(\max\{\widehat f(\lambda),\varepsilon\}\right)^{-1/2},\qquad \widetilde X(\lambda)=W(\lambda)X(\lambda).$$

$$\widehat X_t=\operatorname{IFFT}\left(W(\lambda_k)\operatorname{FFT}(X)_k\right)_t.$$

Mathematical statement

The paper defines the autocovariance of a centered stationary process X_t by gamma(h) = Cov(X_0,X_h), where h is an integer lag, and its spectral density by f(lambda) = (1/(2 pi)) sum_h gamma(h) exp(-i h lambda), where lambda belongs to [-pi, pi]. For a real-valued stationary Gaussian process, f is nonnegative, integrable, and even. We use the corresponding filter W(lambda) = (max(f_hat(lambda), epsilon))^(-1/2), where f_hat is a privatized spectral estimate and epsilon > 0 is a numerical floor. If X has spectrum f, the filtered signal Y = W * X has approximate spectrum f_Y(lambda) = |W(lambda)|^2 f(lambda), which is close to one when f_hat is accurate, thereby flattening temporal correlations. The paper's central privacy result gives an effective sample size N_eff asymptotically proportional to N alpha^4 in the high-privacy regime for dependent Gaussian observations, where N is the number of observations and alpha is the local differential privacy parameter. This scaling determines how strongly the spectrum should be smoothed.

Implementation notes

Integrate this module immediately before the first temporal attention block or SSM, operating independently on each scalar feature channel or on small groups of channels. For a minibatch of length-T sequences, first clip each channel to a fixed range [-B,B]. Compute a periodogram I_b(k) = T^(-1)|FFT(x_b)_k|^2 and aggregate it across examples. For a locally private MVP, privatize clipped cosine and sine autocovariance features with Gaussian local noise or randomized response; after reading the full paper, replace this provisional mechanism with its problem-specific optimal estimator. Form f_hat_k, symmetrize it using (f_hat_k + f_hat_(T-k))/2, project negative values to zero, and smooth across neighboring frequencies. Set N_eff = N alpha^4 and use stronger smoothing when N_eff is small. Compute W_k = (max(f_hat_k, epsilon))^(-1/2), multiply each FFT channel by W_k, and apply an inverse FFT. Keep the filter fixed within an epoch or update it only every several hundred batches to avoid noisy gradients. The paper supplies the Fourier identity and privacy scaling; the engineer estimates f_hat, B, the smoothing bandwidth, and epsilon empirically. Begin with a 2-layer Transformer on synthetic AR(1), long-memory, and white-noise sequences, then test WikiText-2 or a comparable sequence dataset. Compare raw inputs, nonprivate whitening, and private whitening at equal privacy budget. Success means lower validation loss, reduced gradient-norm variance, or matching baseline perplexity with fewer private sequences.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

How to read this: The implementation beat the baseline, but the claimed mechanism did not reproduce at benchmark scale. The gain may come from another effect; treat the cause as unresolved.

Stage 1 — Mechanism check agent confidence 8/10

Built a NumPy MVP for clipped noisy-autocovariance spectral estimation and regularized FFT whitening. The Fourier identity was verified to 1.1e-16 error, exact filtering flattened the spectrum with CV 0.089, and private whitening reduced held-out AR(1) lag-autocorrelation energy from 0.190 to 0.0369 at alpha=1. The effect is real but weaker than nonprivate whitening (0.00150), and stronger privacy degraded spectral MSE substantially.

Agent confidence: 8/10

Baseline: Raw AR(1): lag-autocorrelation energy 0.18995, mean absolute lag-1..10 autocorrelation 0.3561, periodogram CV 2.2066. Nonprivate whitening: 0.001503, 0.0343, 0.2532.
Idea: Private alpha=1 whitening: lag-autocorrelation energy 0.03694, mean absolute lag autocorrelation 0.1374, periodogram CV 0.9082. Alpha=4: 0.02802, 0.1144, 0.7830. Spectral MSE: 11.88 at alpha=0.5, 4.62 at alpha=1, 2.66 at alpha=2, and 2.16 at alpha=4.

Limitations: Only synthetic AR(1) decorrelation was tested; no Transformer or SSM validation loss, gradient variance, language data, long-memory or white-noise benchmarks, GPU path, formal LDP accounting, or rigorous alpha^4 rate fit was included. The Gaussian mechanism is the provisional clipped-feature MVP rather than the paper's optimal estimator.

How to run: python3 private_spectral_whitening.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.24847) Sep 3, 2026

Beats the tuned baseline by 11.3% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.5426
Idea mean: 0.4811
Effect (Δ): -0.0615 (−11.3%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±8.7%
Mechanism: Not confirmed ✗
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in sequence track was tested; no vision, dynamics, language, or custom track was run. The local privacy mechanism is the provisional clipped Gaussian-noise MVP rather than a formally audited optimal LDP estimator. The idea sweep varied learning rate but held alpha=1 and smoothing=5 fixed, and the mechanism signature was measured on one captured seed and did not confirm the predicted decorrelation direction.

How to run: python3 bench_private_whitening.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 3, 2026 latest Worked ✓

Agent confidence: 8/10

Built a local private spectral-whitening front-end and evaluated it end-to-end with the standard sequence forecasting Transformer, using the canonical train_model path and an equal learning-rate sweep. The idea achieved lower test MSE than the tuned baseline on all 8 paired seeds: 0.4811 versus 0.5426, delta -0.06145, permutation p=0.0081, so the harness verdict is idea better (significant). The model-derived mechanism signature was not confirmed: input lag-1 correlation increased rather than decreased after whitening, indicating the accuracy gain is real in this setup but not explained by the intended decorrelation prediction.

Baseline: Sequence track, transformer_tiny, tuned lr=0.0015, 8 epochs: test MSE mean 0.542589, std 0.050570; per-seed [0.581503, 0.460040, 0.474068, 0.612520, 0.567659, 0.546453, 0.579340, 0.519132].
Idea: Private spectral whitening, alpha=1.0, smooth=5, lr=0.0015, 8 epochs: test MSE mean 0.481137, std 0.036006; all 8 paired seeds won; delta -0.061453, permutation p=0.0081. Mechanism signature: baseline lag-1 correlation 0.4115 versus idea 0.8136, confirmed=false.

Benchmark result