# Bayes-bridge parameterization for uniform discrete diffusion

- ID: 432
- Canonical URL: https://synthcore.org/idea/432/bayes-bridge-parameterization-for-uniform-discrete-diffusion
- API JSON: https://synthcore.org/api/idea/432.json
- API Markdown: https://synthcore.org/api/idea/432.md
- Verification status: beats_baseline
- Source: [arXiv:2607.05381](https://arxiv.org/abs/2607.05381)
- Category: architecture
- Solves: stability, accuracy
- ML areas: diffusion, diffusion-sampling, transformer, loss
- Math tags: probability, stochastic-processes, information-theory
- Ratings: usefulness 8/10; difficulty 4/10; novelty 7/10

## Idea description

Train a categorical denoiser for the clean token but convert its output analytically into the reverse CTMC jump rates using the exact forward transition kernel. This separates the easy-to-learn clean-token posterior from the quantity required by the reverse process and should keep the uniform-diffusion ELBO finite at initialization, unlike direct denoiser substitution.

## Mathematical statement

The paper's Oracle Distance theorem states that the negative CTMC ELBO equals the data entropy plus the path KL from the oracle reverse process to the learned reverse process: $-\mathrm{ELBO}(\theta)=H(Z_{0})+\mathrm{KL}_{\mathrm{path}}(R^{\star}\|R_{\theta})$. Here $Z_{0}$ is the clean discrete datum, $Z_{t}$ is its forward-noised state, $H(Z_{0})$ is data entropy, and $R^{\star},R_{\theta}$ are oracle and learned reverse jump-rate processes. The oracle reverse rate from current state $y$ to a different state $x$ is $R^{\star}_{t}(y\to x)=q_{t}(x)Q_{t}(x,y)/q_{t}(y)$, where $q_{t}$ is the marginal probability of $Z_{t}$ and $Q_{t}(x,y)$ is the forward generator rate from $x$ to $y$. Let $d_{t,\theta}(z\mid y)$ be the network's predicted posterior over clean token $z=Z_{0}$ given noisy token $y=Z_{t}$. Bayes' rule gives the exact bridge ratio $A_{t,\theta}(x\mid y)=\sum_{z}d_{t,\theta}(z\mid y)P_{t}(x\mid z)/P_{t}(y\mid z)$, which equals $q_{t}(x)/q_{t}(y)$ for an oracle denoiser. Therefore use $R_{t,\theta}(y\to x)=Q_{t}(x,y)A_{t,\theta}(x\mid y)$ for $x\neq y$. For uniform replacement on an alphabet of size $K$, with total replacement rate $\beta_t$, $Q_t(x,y)=\beta_t/(K-1)$ for $x\neq y$, and $P_t(y\mid z)=\alpha_t\mathbf{1}[y=z]+(1-\alpha_t)/K$, where $\alpha_t=\exp[-K\int_0^t\beta_s/(K-1)\,ds]$.

## Key formulas

- $$-\mathrm{ELBO}(\theta)=H(Z_{0})+\mathrm{KL}_{\mathrm{path}}(R^{\star}\|R_{\theta})$$
- $$R^{\star}_{t}(y\to x)=\frac{q_{t}(x)Q_{t}(x,y)}{q_{t}(y)},\qquad x\neq y$$
- $$A_{t,\theta}(x\mid y)=\sum_{z\in\mathcal A}d_{t,\theta}(z\mid y)\frac{P_{t}(x\mid z)}{P_{t}(y\mid z)},\qquad R_{t,\theta}(y\to x)=Q_{t}(x,y)A_{t,\theta}(x\mid y)$$
- $$P_t(y\mid z)=\alpha_t\mathbf{1}[y=z]+\frac{1-\alpha_t}{K},\quad Q_t(x,y)=\frac{\beta_t}{K-1}\mathbf{1}[x\neq y],\quad \alpha_t=\exp\!\left[-\frac{K}{K-1}\int_0^t\beta_s\,ds\right]$$

## Implementation notes

(1) Integration point: replace the reverse-rate head and sampler of a uniform discrete diffusion transformer. The transformer receives a noisy sequence y and time t, and outputs logits over the clean token z at every position; do not interpret these logits directly as reverse jump rates. (2) Pseudocode: `logits = net(y, t); d = softmax(logits); alpha = exp(-K/(K-1) * integral_beta(t)); for each position i and candidate x != y[i]: for z in vocabulary: P_xz = alpha*(x==z) + (1-alpha)/K; P_yz = alpha*(y[i]==z) + (1-alpha)/K; A[x] = sum_z d[i,z] * P_xz/(P_yz + eps); rate[x] = beta(t)/(K-1) * A[x]; sample a jump using these rates, or use them in the exact CTMC discretization.` The clean-token denoising cross-entropy remains the primary training loss; optionally add a Monte Carlo reverse-rate loss using the computed rates. (3) Computed analytically: $P_t$, $Q_t$, $A_t$, and the reverse rates. Estimated empirically: calibration of $d_{t,\theta}$, the effective ELBO, jump-rate normalization, and sample quality. Use float64 or an epsilon only when $P_t(y\mid z)$ is numerically tiny; for uniform noise it is strictly positive for finite t. (4) First experiment: train a 6-layer transformer on CIFAR-10 flattened into discrete pixel tokens or a small text corpus such as Penn Treebank, comparing direct denoiser-rate substitution against this bridge conversion under identical noise schedules and FLOPs. Measure ELBO at initialization, rate magnitude histograms, NaN/Inf frequency, reverse-process NLL, and sample perplexity or FID. The expected signal is finite and substantially lower initial ELBO, no exploding rates, and improved likelihood or sampling quality at equal training steps, especially for high-noise uniform corruption.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: helps
- Verdict: Built and numerically verified the Bayes-bridge reverse-rate parameterization for uniform categorical diffusion, including the exact kernel, posterior conversion, oracle rates, direct-rate baseline, and CTMC path-KL diagnostic. The mechanism manifested: oracle bridge rates matched oracle rates to 4.4e-16, expected outgoing rate stayed equal to beta across all noise levels, rate error scaled approximately linearly with posterior perturbation, and path KL scaled approximately quadratically. In the toy comparison, bridge KL was substantially lower than direct substitution for moderately accurate denoisers, though it is not uniformly better for a completely uninformative denoiser.

### Mechanism check

- Verdict: Built and numerically verified the Bayes-bridge reverse-rate parameterization for uniform categorical diffusion, including the exact kernel, posterior conversion, oracle rates, direct-rate baseline, and CTMC path-KL diagnostic. The mechanism manifested: oracle bridge rates matched oracle rates to 4.4e-16, expected outgoing rate stayed equal to beta across all noise levels, rate error scaled approximately linearly with posterior perturbation, and path KL scaled approximately quadratically. In the toy comparison, bridge KL was substantially lower than direct substitution for moderately accurate denoisers, though it is not uniformly better for a completely uninformative denoiser.
- Confidence: 9/10
- Limitations: Only a deterministic single-token categorical toy experiment was tested. No transformer training, multi-token sequence modeling, actual ELBO optimization, reverse sampling, likelihood evaluation, GPU benchmark, or sample-quality metric was run; the initialization-divergence claim was not tested as a universal phenomenon because positive softmax outputs avoid literal zero rates.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bayes_bridge_bench.py](https://synthcore.org/code/91/bayes_bridge_bench.py)
- [bench_report.json](https://synthcore.org/code/91/bench_report.json)
- [bridge_experiment.py](https://synthcore.org/code/91/bridge_experiment.py)
- [report.md](https://synthcore.org/code/91/report.md)
- [report_bench_2026-08-30T121614.md](https://synthcore.org/code/91/report_bench_2026-08-30T121614.md)
- [report_bench_2026-08-30T125503.md](https://synthcore.org/code/91/report_bench_2026-08-30T125503.md)
- [report_bench_2026-08-30T144313.md](https://synthcore.org/code/91/report_bench_2026-08-30T144313.md)
- [results.json](https://synthcore.org/code/91/results.json)
- [stage2_bench.py](https://synthcore.org/code/91/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/91)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.