# Fisher-floor-corrected DSM

- ID: 2713
- Canonical URL: https://synthcore.org/idea/2713/fisher-floor-corrected-dsm
- API JSON: https://synthcore.org/api/idea/2713.json
- API Markdown: https://synthcore.org/api/idea/2713.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.23916](https://arxiv.org/abs/2608.23916)
- Category: training
- Solves: accuracy, generalization
- ML areas: diffusion, loss, training
- Math tags: geometry, probability, statistics, information-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace raw denoising score-matching loss reports and weighting decisions with a floor-corrected loss that removes the conditional-target variance intrinsic to the corruption process. This makes models trained under different noise schedules comparable and can produce a lower-variance validation signal for checkpoint selection.

## Mathematical statement

Let Y be a clean data sample, X_t a noisy sample drawn from corruption kernel q_t(x|y), s_theta(x,t) the learned score, s_cond(x,t;y)=∇_x log q_t(x|y) the conditional score, and s(x,t)=∇_x log p_t(x) the marginal score of X_t. Because E[s_cond(X_t,t;Y)|X_t=x]=s(x,t), the conditional-variance decomposition gives E||s_theta(X_t,t)-s_cond(X_t,t;Y)||² = E||s_theta(X_t,t)-s(X_t,t)||² + E tr Cov(s_cond(X_t,t;Y)|X_t). The second term F_t is the irreducible Fisher–Rao floor, where Cov denotes conditional covariance and tr its trace. For Gaussian corruption X_t=α_tY+σ_tε with ε∼N(0,I), s_cond=(α_tY-X_t)/σ_t², so F_t=(α_t²/σ_t⁴)E tr Cov(Y|X_t). The corrected objective subtracts the schedule-dependent floor from the observed DSM loss.

## Key formulas

- $$\mathbf{s}_{\rm cond}(\mathbf{x},t;\mathbf{y})=\nabla_{\mathbf{x}}\log q(\mathbf{x},t\mid\mathbf{y}),\qquad \mathbf{s}(\mathbf{x},t)=\nabla_{\mathbf{x}}\log P_t(\mathbf{x})$$
- $$\mathbb{E}\left[\|\mathbf{s}_\theta(\mathbf{X}_t,t)-\mathbf{s}_{\rm cond}(\mathbf{X}_t,t;\mathbf{Y})\|^2\right]=\mathbb{E}\left[\|\mathbf{s}_\theta(\mathbf{X}_t,t)-\mathbf{s}(\mathbf{X}_t,t)\|^2\right]+\mathbb{E}\left[\operatorname{tr}\operatorname{Cov}(\mathbf{s}_{\rm cond}(\mathbf{X}_t,t;\mathbf{Y})\mid\mathbf{X}_t)\right]$$
- $$X_t=\alpha_tY+\sigma_t\varepsilon,\quad \mathbf{s}_{\rm cond}(x,t;y)=\frac{\alpha_t y-x}{\sigma_t^2},\quad F_t=\frac{\alpha_t^2}{\sigma_t^4}\,\mathbb{E}\left[\operatorname{tr}\operatorname{Cov}(Y\mid X_t)\right]$$
- $$\mathcal{L}_{\rm corr}=\int w(t)\left(\mathbb{E}\|s_\theta(X_t,t)-s_{\rm cond}(X_t,t;Y)\|^2-F_t\right)\,dt$$

## Implementation notes

(1) Integrate this at diffusion-model validation and loss-weighting time, immediately after computing the per-example DSM squared error ||s_theta(x_t,t)-s_cond||². Initially leave the score network unchanged and use the same noise schedule and time weighting as the baseline. For Gaussian corruption x=alpha*y+sigma*epsilon, estimate the floor on each minibatch using an empirical clean-data prior. For each noisy query x_i and bank sample y_j, compute logits ell_ij=-||x_i-alpha*y_j||²/(2*sigma²), posterior weights pi_ij=softmax_j(ell_ij), posterior mean mu_i=sum_j pi_ij*y_j, and posterior covariance trace v_i=sum_j pi_ij||y_j-mu_i||². Set Fhat_i=(alpha²/sigma⁴)v_i and subtract the minibatch mean of Fhat from the raw loss; stop gradients through Fhat. For high-dimensional data, use a random memory bank and compute only the covariance trace, not the full matrix. Pseudocode: sample y,t,epsilon; form x=alpha(t)y+sigma(t)epsilon; target=(alpha*y-x)/sigma²; raw=||net(x,t)-target||²; compute pi over the bank; floor=alpha²/sigma⁴ times weighted variance; optimize mean(w(t)*(raw-floor)). The decomposition and Gaussian floor formula come from the mathematics; the finite-bank posterior is an approximation. First test on MNIST or CIFAR-10 with small DDPMs under cosine and linear schedules. Compare raw loss, corrected loss, and FID or held-out score error. Success means corrected loss ranks checkpoints and schedules more consistently with sample quality and removes schedule-dependent additive offsets.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built and ran a reproducible Fisher-floor DSM toy experiment for a 1D Gaussian-mixture prior with exact posterior floors and a finite memory-bank estimator. The decomposition residual was 0.39%, alpha scaling matched the predicted quadratic law within 4.7%, alpha=0 gave exactly zero floor, and bank error scaled with slope -0.461 versus the predicted -0.5. Pooled checkpoint rank correlation with ideal score error improved from 0.615 for raw DSM to 0.937 for corrected DSM, while the cross-schedule loss offset fell from 1.680 to -0.019; this supports schedule comparability but does not establish improved sample quality.

### Mechanism check

- Verdict: Built and ran a reproducible Fisher-floor DSM toy experiment for a 1D Gaussian-mixture prior with exact posterior floors and a finite memory-bank estimator. The decomposition residual was 0.39%, alpha scaling matched the predicted quadratic law within 4.7%, alpha=0 gave exactly zero floor, and bank error scaled with slope -0.461 versus the predicted -0.5. Pooled checkpoint rank correlation with ideal score error improved from 0.615 for raw DSM to 0.937 for corrected DSM, while the cross-schedule loss offset fell from 1.680 to -0.019; this supports schedule comparability but does not establish improved sample quality.
- Confidence: 9/10
- Limitations: Only a low-dimensional synthetic Gaussian-mixture experiment was tested. No MNIST/CIFAR diffusion network, training-speed or FLOP comparison, FID, or real checkpoint-selection experiment was performed. The finite-bank estimator has sampling noise and can produce slightly negative corrected empirical losses.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 1
- Paired wins: 0/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_exp.py](https://synthcore.org/code/1046/bench_exp.py)
- [bench_report.json](https://synthcore.org/code/1046/bench_report.json)
- [fisher_floor_dsm.py](https://synthcore.org/code/1046/fisher_floor_dsm.py)
- [fisher_floor_experiment.py](https://synthcore.org/code/1046/fisher_floor_experiment.py)
- [report.md](https://synthcore.org/code/1046/report.md)
- [report_bench_2026-09-01T211550.md](https://synthcore.org/code/1046/report_bench_2026-09-01T211550.md)
- [results.json](https://synthcore.org/code/1046/results.json)
- [verify_floor_dsm.py](https://synthcore.org/code/1046/verify_floor_dsm.py)
- [Download all files as ZIP](https://synthcore.org/download/1046)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.