# Intrinsic-Dimension Batch Audit

- ID: 487
- Canonical URL: https://synthcore.org/idea/487/intrinsic-dimension-batch-audit
- API JSON: https://synthcore.org/api/idea/487.json
- API Markdown: https://synthcore.org/api/idea/487.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.06644](https://arxiv.org/abs/2607.06644)
- Category: regularization
- Solves: accuracy, stability, sample-efficiency
- ML areas: training, optimizer, regularization
- Math tags: geometry, graph-theory, spectral-theory, probability
- Ratings: usefulness 6/10; difficulty 7/10; novelty 7/10

## Idea description

Use the diffusion graph's Dirichlet energy and almost-isometry inequalities to score whether a candidate minibatch preserves the low-frequency structure of losses, logits, or gradients over the dataset. Reject or augment batches that distort these quantities, producing a geometry-aware batch acceptance rule rather than relying only on random or loss-based sampling.

## Mathematical statement

For a data function $f$ defined on the full dataset, let $D_{1-2/m}(f)$ denote its fractional continuum Dirichlet energy, $b_k(u)$ the corresponding k-NN graph Dirichlet energy for a discrete vector $u$, and $\widetilde P f$ the restriction or interpolation between continuum and graph samples. Proposition 5.12 gives the uniform almost-isometry estimate $|\|f\|_{L^2(\mu)}^2-\|\widetilde P f\|_{L^2(\mu_N)}^2|\leq C\widetilde\delta\|f\|_{L^2(\mu)}\sqrt{D_{1-2/m}(f)}+C(\theta+\widetilde\delta)\|f\|_{L^2(\mu)}^2$, where $\mu$ is the population measure, $\mu_N$ is the empirical measure on $N$ points, $\widetilde\delta$ is the continuum-to-sample scale, $\theta$ is a concentration parameter, and $C$ is independent of $f$. The discrete analogue is $|\|u\|_{L^2(\mu_N)}^2-\|\widetilde{\mathcal I}u\|_{L^2(\mu)}^2|\leq C(k/N)^{1/m}\|u\|_{L^2(\mu_N)}\sqrt{b_k(u)}+C(\theta+\widetilde\delta)\|u\|_{L^2(\mu_N)}^2$, where $\widetilde{\mathcal I}$ interpolates graph values and $m$ is intrinsic dimension. In a neural network, estimate analogous norm and energy distortions on a probe set and use them as a batch-quality criterion.

## Key formulas

- $$\left|\lVert f\rVert_{L^{2}(\mu)}^{2}-\lVert\widetilde{P}f\rVert_{L^{2}(\mu_{N})}^{2}\right|\leq C\widetilde{\delta}\lVert f\rVert_{L^{2}(\mu)}\sqrt{D_{1-2/m}(f)}+C(\theta+\widetilde{\delta})\lVert f\rVert_{L^{2}(\mu)}^{2}$$
- $$\left|\lVert u\rVert_{L^{2}(\mu_{N})}^{2}-\lVert\widetilde{\mathcal{I}}u\rVert_{L^{2}(\mu)}^{2}\right|\leq C(k/N)^{1/m}\lVert u\rVert_{L^{2}(\mu_{N})}\sqrt{b_k(u)}+C(\theta+\widetilde{\delta})\lVert u\rVert_{L^{2}(\mu_{N})}^{2}$$
- $$Q(S)=\frac{\left|\widehat{\|f\|^2}_S-\widehat{\|f\|^2}_{\mathrm{probe}}\right|}{\widehat{\|f\|^2}_{\mathrm{probe}}+\epsilon}+\alpha\frac{\left|\widehat{b_k(f)}_S-b_k(f_{\mathrm{probe}})\right|}{b_k(f_{\mathrm{probe}})+\epsilon}$$

## Implementation notes

1. Integration point: add a lightweight batch-quality controller around the dataloader. At each validation interval, evaluate the current model's scalar loss, one or a few gradient projections, and selected logit coordinates on a fixed probe set of $M$ examples. Treat each scalar over the dataset as a function $f$. Build or reuse the frozen k-NN graph and store neighbor lists and edge weights. 2. Pseudocode: draw a candidate batch $S$ using uniform sampling or the diffusion-DPP sampler; evaluate probe values $f_i$ and batch values; compute the weighted batch norm estimate $\widehat{\|f\|^2}_S=N^{-1}\sum_{i\in S}f_i^2/\pi_i$ and graph energy $\widehat b_k(f)_S$ using edges incident to sampled points with Horvitz-Thompson edge weights; compute $Q(S)$ from the displayed distortion formula; accept if $Q(S)\leq\tau$, otherwise draw another candidate or append high-scoring uncovered points. Maintain an exponential moving average of $Q$ and stop rejection after two retries. 3. The paper's formulas determine the two distortion terms and explain why low-energy functions should be preserved; constants $C$, $\widetilde\delta$, and $\theta$ need not be known. Estimate baseline norms and energies empirically from the probe set, estimate $m$ from a log-log fit of graph Laplacian eigenvalues, and tune $\alpha$ and $\tau$ on a held-out run. Do not compute full gradients for every candidate: use loss and random gradient projections first, then periodically audit the true gradient. 4. First cheap experiment: train a small MLP or ResNet-18 on CIFAR-10 with batch size 64. Compare uniform batches, DPP batches without auditing, and DPP plus the acceptance rule at equal attempted examples. Track full-dataset gradient cosine similarity every 500 steps, batch-loss bias, gradient-norm spikes, validation accuracy, and rejected-batch overhead. Success means improved gradient cosine similarity and reduced loss-estimate bias at the same batch size, with less than 20% dataloader overhead and no increase in optimizer instability.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a reproducible intrinsic-dimension batch audit prototype with k-NN graph energy, norm/energy distortion scoring, candidate retries, and a small MLP comparison. The toy math check showed graph energy increasing with frequency and auditing reduced norm-estimation error for frequencies 1–8, but the training experiment showed no gradient-cosine or loss-bias improvement: cosine was 0.3318 baseline versus 0.3294 audited, and bias was 0.01246 versus 0.01247. The audited controller also evaluated three candidates per update, giving 11.7% measured local overhead but 3x attempted examples and lower accuracy, so the promised practical win was not observed.

### Mechanism check

- Verdict: Built a reproducible intrinsic-dimension batch audit prototype with k-NN graph energy, norm/energy distortion scoring, candidate retries, and a small MLP comparison. The toy math check showed graph energy increasing with frequency and auditing reduced norm-estimation error for frequencies 1–8, but the training experiment showed no gradient-cosine or loss-bias improvement: cosine was 0.3318 baseline versus 0.3294 audited, and bias was 0.01246 versus 0.01247. The audited controller also evaluated three candidates per update, giving 11.7% measured local overhead but 3x attempted examples and lower accuracy, so the promised practical win was not observed.
- Confidence: 8/10
- Limitations: This was a small synthetic circle-manifold experiment rather than CIFAR-10, used scalar losses and one logit coordinate instead of true gradient projections, and computed full-dataset audit values for convenience. It did not implement diffusion-DPP sampling, arbitrary-inclusion-probability Horvitz-Thompson edge handling, probe refresh intervals, threshold tuning, or a replicated held-out study.

## Artifacts

- [audit_experiment.py](https://synthcore.org/code/1356/audit_experiment.py)
- [report.md](https://synthcore.org/code/1356/report.md)
- [results.json](https://synthcore.org/code/1356/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1356)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.