Intrinsic-Dimension Batch Audit
Implementation & benchmark of arXiv:2607.06644 — Fast determinantal sampling on general spaces and diffusion geometry
Source paper: Fast determinantal sampling on general spaces and diffusion geometry arXiv:2607.06644 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Use the diffusion graph's Dirichlet energy and almost-isometry inequalities to score whether a candidate minibatch preserves the low-frequency structure of losses, logits, or gradients over the dataset. Reject or augment batches that distort these quantities, producing a geometry-aware batch acceptance rule rather than relying only on random or loss-based sampling.
Formulas
Mathematical statement
For a data function $f$ defined on the full dataset, let $D_{1-2/m}(f)$ denote its fractional continuum Dirichlet energy, $b_k(u)$ the corresponding k-NN graph Dirichlet energy for a discrete vector $u$, and $\widetilde P f$ the restriction or interpolation between continuum and graph samples. Proposition 5.12 gives the uniform almost-isometry estimate $|\|f\|_{L^2(\mu)}^2-\|\widetilde P f\|_{L^2(\mu_N)}^2|\leq C\widetilde\delta\|f\|_{L^2(\mu)}\sqrt{D_{1-2/m}(f)}+C(\theta+\widetilde\delta)\|f\|_{L^2(\mu)}^2$, where $\mu$ is the population measure, $\mu_N$ is the empirical measure on $N$ points, $\widetilde\delta$ is the continuum-to-sample scale, $\theta$ is a concentration parameter, and $C$ is independent of $f$. The discrete analogue is $|\|u\|_{L^2(\mu_N)}^2-\|\widetilde{\mathcal I}u\|_{L^2(\mu)}^2|\leq C(k/N)^{1/m}\|u\|_{L^2(\mu_N)}\sqrt{b_k(u)}+C(\theta+\widetilde\delta)\|u\|_{L^2(\mu_N)}^2$, where $\widetilde{\mathcal I}$ interpolates graph values and $m$ is intrinsic dimension. In a neural network, estimate analogous norm and energy distortions on a probe set and use them as a batch-quality criterion.
Implementation notes
1. Integration point: add a lightweight batch-quality controller around the dataloader. At each validation interval, evaluate the current model's scalar loss, one or a few gradient projections, and selected logit coordinates on a fixed probe set of $M$ examples. Treat each scalar over the dataset as a function $f$. Build or reuse the frozen k-NN graph and store neighbor lists and edge weights. 2. Pseudocode: draw a candidate batch $S$ using uniform sampling or the diffusion-DPP sampler; evaluate probe values $f_i$ and batch values; compute the weighted batch norm estimate $\widehat{\|f\|^2}_S=N^{-1}\sum_{i\in S}f_i^2/\pi_i$ and graph energy $\widehat b_k(f)_S$ using edges incident to sampled points with Horvitz-Thompson edge weights; compute $Q(S)$ from the displayed distortion formula; accept if $Q(S)\leq\tau$, otherwise draw another candidate or append high-scoring uncovered points. Maintain an exponential moving average of $Q$ and stop rejection after two retries. 3. The paper's formulas determine the two distortion terms and explain why low-energy functions should be preserved; constants $C$, $\widetilde\delta$, and $\theta$ need not be known. Estimate baseline norms and energies empirically from the probe set, estimate $m$ from a log-log fit of graph Laplacian eigenvalues, and tune $\alpha$ and $\tau$ on a held-out run. Do not compute full gradients for every candidate: use loss and random gradient projections first, then periodically audit the true gradient. 4. First cheap experiment: train a small MLP or ResNet-18 on CIFAR-10 with batch size 64. Compare uniform batches, DPP batches without auditing, and DPP plus the acceptance rule at equal attempted examples. Track full-dataset gradient cosine similarity every 500 steps, batch-loss bias, gradient-norm spikes, validation accuracy, and rejected-batch overhead. Success means improved gradient cosine similarity and reduced loss-estimate bias at the same batch size, with less than 20% dataloader overhead and no increase in optimizer instability.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 8/10
Built a reproducible intrinsic-dimension batch audit prototype with k-NN graph energy, norm/energy distortion scoring, candidate retries, and a small MLP comparison. The toy math check showed graph energy increasing with frequency and auditing reduced norm-estimation error for frequencies 1–8, but the training experiment showed no gradient-cosine or loss-bias improvement: cosine was 0.3318 baseline versus 0.3294 audited, and bias was 0.01246 versus 0.01247. The audited controller also evaluated three candidates per update, giving 11.7% measured local overhead but 3x attempted examples and lower accuracy, so the promised practical win was not observed.
- Agent confidence
- 8/10
- Baseline
- gradient cosine 0.3318; loss-estimate bias 0.01246; final-window loss 0.6851; accuracy 0.6139; retries 0; audit overhead 4.1%
- Idea
- gradient cosine 0.3294; loss-estimate bias 0.01247; final-window loss 0.6760; accuracy 0.5431; retries 2; audit overhead 11.7%; audited norm error was lower than random for frequencies 1–8
Limitations: This was a small synthetic circle-manifold experiment rather than CIFAR-10, used scalar losses and one logit coordinate instead of true gradient projections, and computed full-dataset audit values for convenience. It did not implement diffusion-DPP sampling, arbitrary-inclusion-probability Horvitz-Thompson edge handling, probe refresh intervals, threshold tuning, or a replicated held-out study.
How to run: python3 audit_experiment.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Artifacts
- audit_experiment.py 5.2 KB View Raw
- report.md 1.7 KB View
- results.json 1.7 KB View Raw