△ Mechanism confirmed, baseline not beaten 2026

Fisher-floor-corrected DSM

Implementation & benchmark of arXiv:2608.23916 — The Loss Floor of Denoising Score Matching: Fisher Geometry from Schrödinger Bridges

Usefulness7/10

Difficulty5/10

Novelty7/10

Training Accuracy Generalization Diffusion Loss Training Geometry Probability Statistics Information theory

Source paper: The Loss Floor of Denoising Score Matching: Fisher Geometry from Schrödinger Bridges arXiv:2608.23916 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace raw denoising score-matching loss reports and weighting decisions with a floor-corrected loss that removes the conditional-target variance intrinsic to the corruption process. This makes models trained under different noise schedules comparable and can produce a lower-variance validation signal for checkpoint selection.

Formulas

$$\mathbf{s}_{\rm cond}(\mathbf{x},t;\mathbf{y})=\nabla_{\mathbf{x}}\log q(\mathbf{x},t\mid\mathbf{y}),\qquad \mathbf{s}(\mathbf{x},t)=\nabla_{\mathbf{x}}\log P_t(\mathbf{x})$$

$$\mathbb{E}\left[\|\mathbf{s}_\theta(\mathbf{X}_t,t)-\mathbf{s}_{\rm cond}(\mathbf{X}_t,t;\mathbf{Y})\|^2\right]=\mathbb{E}\left[\|\mathbf{s}_\theta(\mathbf{X}_t,t)-\mathbf{s}(\mathbf{X}_t,t)\|^2\right]+\mathbb{E}\left[\operatorname{tr}\operatorname{Cov}(\mathbf{s}_{\rm cond}(\mathbf{X}_t,t;\mathbf{Y})\mid\mathbf{X}_t)\right]$$

$$X_t=\alpha_tY+\sigma_t\varepsilon,\quad \mathbf{s}_{\rm cond}(x,t;y)=\frac{\alpha_t y-x}{\sigma_t^2},\quad F_t=\frac{\alpha_t^2}{\sigma_t^4}\,\mathbb{E}\left[\operatorname{tr}\operatorname{Cov}(Y\mid X_t)\right]$$

$$\mathcal{L}_{\rm corr}=\int w(t)\left(\mathbb{E}\|s_\theta(X_t,t)-s_{\rm cond}(X_t,t;Y)\|^2-F_t\right)\,dt$$

Mathematical statement

Let Y be a clean data sample, X_t a noisy sample drawn from corruption kernel q_t(x|y), s_theta(x,t) the learned score, s_cond(x,t;y)=∇_x log q_t(x|y) the conditional score, and s(x,t)=∇_x log p_t(x) the marginal score of X_t. Because E[s_cond(X_t,t;Y)|X_t=x]=s(x,t), the conditional-variance decomposition gives E||s_theta(X_t,t)-s_cond(X_t,t;Y)||² = E||s_theta(X_t,t)-s(X_t,t)||² + E tr Cov(s_cond(X_t,t;Y)|X_t). The second term F_t is the irreducible Fisher–Rao floor, where Cov denotes conditional covariance and tr its trace. For Gaussian corruption X_t=α_tY+σ_tε with ε∼N(0,I), s_cond=(α_tY-X_t)/σ_t², so F_t=(α_t²/σ_t⁴)E tr Cov(Y|X_t). The corrected objective subtracts the schedule-dependent floor from the observed DSM loss.

Implementation notes

(1) Integrate this at diffusion-model validation and loss-weighting time, immediately after computing the per-example DSM squared error ||s_theta(x_t,t)-s_cond||². Initially leave the score network unchanged and use the same noise schedule and time weighting as the baseline. For Gaussian corruption x=alpha*y+sigma*epsilon, estimate the floor on each minibatch using an empirical clean-data prior. For each noisy query x_i and bank sample y_j, compute logits ell_ij=-||x_i-alpha*y_j||²/(2*sigma²), posterior weights pi_ij=softmax_j(ell_ij), posterior mean mu_i=sum_j pi_ij*y_j, and posterior covariance trace v_i=sum_j pi_ij||y_j-mu_i||². Set Fhat_i=(alpha²/sigma⁴)v_i and subtract the minibatch mean of Fhat from the raw loss; stop gradients through Fhat. For high-dimensional data, use a random memory bank and compute only the covariance trace, not the full matrix. Pseudocode: sample y,t,epsilon; form x=alpha(t)y+sigma(t)epsilon; target=(alpha*y-x)/sigma²; raw=||net(x,t)-target||²; compute pi over the bank; floor=alpha²/sigma⁴ times weighted variance; optimize mean(w(t)*(raw-floor)). The decomposition and Gaussian floor formula come from the mathematics; the finite-bank posterior is an approximation. First test on MNIST or CIFAR-10 with small DDPMs under cosine and linear schedules. Compare raw loss, corrected loss, and FID or held-out score error. Success means corrected loss ranks checkpoints and schedules more consistently with sample quality and removes schedule-dependent additive offsets.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built and ran a reproducible Fisher-floor DSM toy experiment for a 1D Gaussian-mixture prior with exact posterior floors and a finite memory-bank estimator. The decomposition residual was 0.39%, alpha scaling matched the predicted quadratic law within 4.7%, alpha=0 gave exactly zero floor, and bank error scaled with slope -0.461 versus the predicted -0.5. Pooled checkpoint rank correlation with ideal score error improved from 0.615 for raw DSM to 0.937 for corrected DSM, while the cross-schedule loss offset fell from 1.680 to -0.019; this supports schedule comparability but does not establish improved sample quality.

Agent confidence: 9/10

Baseline: Raw DSM pooled checkpoint-vs-ideal rank correlation 0.615; mean cross-schedule loss offset 1.680.
Idea: Corrected DSM pooled rank correlation 0.937; mean cross-schedule offset -0.019; decomposition relative error 0.0039; alpha quadratic-fit maximum relative error 0.0468; bank log-log error slope -0.461 versus -0.5 predicted; alpha=0 floor 0.0.

Limitations: Only a low-dimensional synthetic Gaussian-mixture experiment was tested. No MNIST/CIFAR diffusion network, training-speed or FLOP comparison, FID, or real checkpoint-selection experiment was performed. The finite-bank estimator has sampling noise and can produce slightly negative corrected empirical losses.

How to run: python3 fisher_floor_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.23916) Sep 2, 2026

No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 8 paired seeds).

Benchmark: Multitoken_diffusion
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 22.1827
Idea mean: 22.1827
Effect (Δ): 0 (−-0.0%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 1 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.0%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

One registered synthetic multi-token diffusion track and one mlp_tiny architecture were tested. Checkpoint ranking, multiple corruption schedules, FID/sample quality, larger memory banks, and non-detached floors or floor-based timestep reweighting were not tested.

How to run: python3 bench_exp.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

Re-ran the full registered multitoken_diffusion benchmark with 8 paired seeds, a tuned baseline sweep, and a three-setting idea sweep. The corrected DSM system exactly matched raw DSM on every paired seed (delta_mean=0, p=1.0), so there was no significant win. This is expected because detached floor subtraction is parameter-independent and leaves gradients unchanged under identical timestep weighting.

Baseline: multitoken_diffusion / mlp_tiny; best lr=0.003, epochs=15; test MSE mean 22.182694673538208, std 4.590790464181882.
Idea: Best lr=0.003, epochs=15; test MSE mean 22.182694673538208, std 4.590790464181882; paired delta 0.0; p=1.0; verdict no measurable effect.

Benchmark result