△ Mechanism confirmed, baseline not beaten 2026

Convex-Projected Diffusion Sampler

Implementation & benchmark of arXiv:2607.08757 — Score Accuracy Along the Forward Diffusion Does Not Certify Numerical Stability in Diffusion Sampling

Usefulness8/10
Difficulty3/10
Novelty7/10

Source paper: Score Accuracy Along the Forward Diffusion Does Not Certify Numerical Stability in Diffusion Sampling arXiv:2607.08757 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

For additive-noise diffusion, train or interpret the network as a denoiser and project its predicted clean sample onto a known bounded closed convex set containing the data support. Convert the projected denoiser back into a score before each Euler-Maruyama or probability-flow ODE step. The projection is nonexpansive relative to the true denoiser, so it cannot increase pointwise denoising error when the true conditional mean belongs to the set, while it imposes a hard bound that suppresses rare numerical explosions.

Formulas

$$\mathcal{E}_{\mathrm{score}}=\int_{\delta}^{T}\mathbb{E}_{X_{t}\sim p_{t}}\|\widehat{s}_{t}(X_{t})-s_{t}(X_{t})\|^{2}\,dt.$$
$$X_t=X_0+\sigma_t\varepsilon,\qquad m_t(x)=\mathbb{E}[X_0\mid X_t=x],\qquad s_t(x)=\frac{m_t(x)-x}{\sigma_t^2}.$$
$$\|\Pi_K(a)-z\|\leq\|a-z\|\quad(z\in K),\qquad \widetilde m_t(x)=\Pi_K(\widehat m_t(x)),\qquad \widetilde s_t(x)=\frac{\widetilde m_t(x)-x}{\sigma_t^2}.$$
$$x_{k-1}=x_k+h\,b_{\mathrm{rev}}(x_k,t_k;\widetilde s_{t_k})+g(t_k)\sqrt{h}\,\xi_k,\qquad \xi_k\sim\mathcal N(0,I).$$

Mathematical statement

The paper studies the on-path score error $\mathcal{E}_{\mathrm{score}}=\int_{\delta}^{T}\mathbb{E}_{X_t\sim p_t}\|\widehat{s}_t(X_t)-s_t(X_t)\|^2\,dt$, where $p_t$ is the forward marginal, $s_t$ is the exact score, and $\widehat{s}_t$ is the learned score. This average error does not control the learned score on rare states visited by the discretized reverse process. For an additive Gaussian forward diffusion $X_t=X_0+\sigma_t\varepsilon$, define the clean-data denoiser $m_t(x)=\mathbb{E}[X_0\mid X_t=x]$ and the score relation $s_t(x)=(m_t(x)-x)/\sigma_t^2$, with noise scale $\sigma_t>0$. If the clean data satisfy $X_0\in K$ almost surely for a bounded closed convex set $K$, then $m_t(x)\in K$ because it is a conditional expectation of points in $K$. Let $\Pi_K$ be Euclidean projection onto $K$. The projection theorem gives $\|\Pi_K(a)-z\|\leq\|a-z\|$ for every $z\in K$, hence $\|\Pi_K(\widehat m_t(x))-m_t(x)\|\leq\|\widehat m_t(x)-m_t(x)\|$. It also gives $\|\Pi_K(\widehat m_t(x))\|\leq R$ when $K\subseteq\{z:\|z\|\leq R\}$, preventing arbitrarily large denoiser values. Use the adapted score $\widetilde s_t(x)=(\Pi_K(\widehat m_t(x))-x)/\sigma_t^2$ in the reverse sampler.

Implementation notes

Use a VE diffusion sampler first, because the denoiser-to-score relation is explicit. Let the network output $\widehat m_\theta(x,t)$, either by retraining the final head with denoising-score matching or by converting an existing score prediction using $\widehat m_\theta=x+\sigma_t^2\widehat s_\theta$. Choose a bounded closed convex set $K$: for normalized images, an elementwise box $[-1,1]^d$ is the cheapest baseline; for vector data use a ball or a tighter coordinate box estimated from the training set with a safety margin. At every reverse-time grid point, compute the denoiser, project it, convert it to a score, and pass that score to the existing reverse drift. Pseudocode: x = sample_prior(); for k=T,...,1: m_hat = net(x,t[k]); m_proj = project_K(m_hat); s = (m_proj-x)/sigma(t[k])**2; x = x + h*reverse_drift(x,t[k],s) + g(t[k])*sqrt(h)*randn(); return x. The mathematical guarantee is the projection inequality and the bounded-output property; estimate $K$, score calibration, and numerical effects empirically. First test on CIFAR-10 with a small DiT or U-Net, comparing unprojected Euler-Maruyama, projected Euler-Maruyama, and deterministic probability-flow sampling at identical networks and step counts. Track FID, precision/recall, maximum denoiser norm, fraction of trajectories exceeding a large state norm, endpoint $W_1$ against a reference fine-step sampler, and how these quantities change with step size. Success means substantially fewer rare large-norm trajectories and bounded endpoint moments, with no meaningful FID degradation; a stronger result is improved FID or Wasserstein error at coarse step counts.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built and ran a reproducible scalar VE/probability-flow Euler MVP for convex-projected denoising. All three mechanism predictions were confirmed: zero pointwise projection-error violations, an exact projected-denoiser bound, and monotone raw-tail instability with stable projected trajectories. This demonstrates the proposed mechanism, but not a general image-generation win.

Agent confidence
9/10
Baseline
Unprojected Euler: mean maximum state increased from 1.328 to 3.014 and endpoint W1 versus the zero-tail reference increased from 0 to 1.761 as tail amplitude rose from 0 to 40; 5.58% of trajectories exceeded state norm 10 at amplitude 40.
Idea
Projected Euler: mean maximum state stayed at approximately 1.328 across amplitudes, zero trajectories exceeded norm 10, projected denoiser maximum was exactly 1.0, and endpoint W1 stayed below 6.5e-5.

Limitations: Only a 1D controlled distribution and intentionally adversarial tail error were tested. No trained neural network, CIFAR/image metric, stochastic Euler-Maruyama comparison, runtime overhead, or misspecified convex-set experiment was evaluated.

How to run: python3 toy_verify.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.08757)

Significantly WORSE than the tuned baseline (+6.0%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark
Uniform discrete diffusion (multi-token)
Model
transformer_tiny
Paired seeds
8
Baseline mean
0.675
Idea mean
0.7156
Effect (Δ)
0.0406 (+6.0%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect
±2.3%
Mechanism
Confirmed ✓
Practical effect
Hurts
Baseline tuning
swept over 3 configs
Limitations:

The registered benchmark was uniform_diffusion_multitoken rather than CIFAR-10 image sampling. The categorical track required a continuous denoiser expectation followed by projection and conversion back to logits; reverse Euler-Maruyama or probability-flow endpoint quality, FID, runtime overhead, and misspecified convex supports were not tested.

How to run: python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark latest Failed ✗

Agent confidence: 10/10

The convex-projected system was significantly worse than the unprojected baseline on the registered multitoken diffusion track. The mechanism signature was confirmed, but the independent benchmark metric degraded: delta_mean=+0.04060547 with permutation p=0.0081, so this is not a Stage-2 win.

Baseline
Best lr=0.003, epochs=6; mean metric 0.6750390455 across 8 seeds.
Idea
Best projected lr=0.0015, epochs=6; mean metric 0.7156445161 across 8 seeds.

Benchmark result

Significantly WORSE than the tuned baseline (+6.0%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark
Uniform discrete diffusion (multi-token)
Model
transformer_tiny
Paired seeds
8
Baseline mean
0.675
Idea mean
0.7156
Effect (Δ)
0.0406 (+6.0%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect
±2.3%
Mechanism
Confirmed ✓
Practical effect
Hurts
Baseline tuning
swept over 3 configs
Limitations:

The registered benchmark was uniform_diffusion_multitoken rather than CIFAR-10 image sampling. The categorical track required a continuous denoiser expectation followed by projection and conversion back to logits; reverse Euler-Maruyama or probability-flow endpoint quality, FID, runtime overhead, and misspecified convex supports were not tested.

How to run: python3 bench_run.py

Artifacts

Implementation overview ⬇ Download all as ZIP 9 files · code, reports and structured results