# Bayesian Logit Smoother with Bursty-Mask Marginalization

- ID: 2645
- Canonical URL: https://synthcore.org/idea/2645/bayesian-logit-smoother-with-bursty-mask-marginalization
- API JSON: https://synthcore.org/api/idea/2645.json
- API Markdown: https://synthcore.org/api/idea/2645.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.21132](https://arxiv.org/abs/2608.21132)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rnn, ssm, training-dynamics, regularization
- Math tags: probability, stochastic-processes, dynamical-systems, statistics
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Attach a recursive Bayesian state estimator to a neural sequence classifier. The network produces per-step emission likelihoods, while a persistent Markov transition model propagates beliefs between steps; when inputs are missing, marginalize the missing emission instead of replacing it with a sentinel or arbitrary imputation. This should suppress isolated logit oscillations and remain robust when missing data arrive in bursts.

## Mathematical statement

Let $z_t\in\{1,\ldots,K\}$ be the latent class at time $t$, $x_t$ the observed input, $e_t(s)=p_\theta(x_t\mid z_t=s)$ the neural emission likelihood, $A_{rs}=p(z_t=s\mid z_{t-1}=r)$ the class-transition matrix, and $b_t(s)=p(z_t=s\mid x_{1:t})$ the filtered belief. The recursive Bayesian update is $\tilde b_t(s)=e_t(s)\sum_{r=1}^{K}A_{rs}b_{t-1}(r)$ followed by normalization $b_t(s)=\tilde b_t(s)/\sum_j\tilde b_t(j)$. For a missing observation, use $e_t(s)=1$, so the update becomes pure Markov prediction. To model bursty missingness, let $m_t\in\{0,1\}$ denote observed versus dropped input and use the transition matrix $M=\begin{bmatrix}1-\rho/((1-\rho)b)&\rho/((1-\rho)b)\\1/b&1-1/b\end{bmatrix}$ for states present and dropped, where $\rho$ is the stationary dropout probability and $b$ is the mean dropped-burst length. The dropped-state persistence eigenvalue is approximately $1-1/b$, giving a predicted belief memory time of $\tau\approx-b/\log(1-1/b)\approx b$ frames. If the class transition matrix has self-transition probability $a$ and off-diagonal probability $(1-a)/(K-1)$, the nontrivial belief mode contracts by $\lambda_A=a-(1-a)/(K-1)$ per step; hence temporal perturbations should decay as $|\lambda_A|^q$ after $q$ steps without informative emissions.

## Key formulas

- $$\tilde b_t(s)=e_t(s)\sum_{r=1}^{K}A_{rs}b_{t-1}(r),\qquad b_t(s)=\frac{\tilde b_t(s)}{\sum_{j=1}^{K}\tilde b_t(j)}.$$
- $$e_t(s)=\begin{cases}\exp\!\bigl(\ell_\theta(x_t,s)\bigr),&m_t=1,\\1,&m_t=0,\end{cases}$$
- $$M=\begin{bmatrix}1-\frac{\rho}{(1-\rho)b}&\frac{\rho}{(1-\rho)b}\\[2pt]\frac{1}{b}&1-\frac{1}{b}\end{bmatrix},\qquad \pi_M=[1-\rho,\rho].$$
- $$\|\delta b_{t+q}\|\lesssim |\lambda_A|^q\|\delta b_t\|,\qquad \lambda_A=a-\frac{1-a}{K-1}.$$

## Implementation notes

1. Integration point: use any framewise neural classifier, such as an MLP, CNN encoder, transformer encoder, or SSM, producing logits $\ell_\theta(x_t)\in\mathbb{R}^K$. Convert logits to positive emissions with $e_t(s)=\exp(\ell_\theta(x_t,s)-\operatorname{logsumexp}_j\ell_\theta(x_t,j))$; equivalently these are normalized neural likelihood scores. Maintain a belief vector $b$ outside the network and expose $b_t$ as the final prediction or feed it into a downstream recurrent policy. 2. Pseudocode: initialize $b=\pi_A$; for every timestep, compute the neural logits and mask $m_t$; set $e=\operatorname{softmax}(\ell)$ if observed and $e=\mathbf{1}$ if missing; compute $p=A^\top b$; update $b\leftarrow\operatorname{normalize}(e\odot p)$; return $b$. Train with cross-entropy on $b_t$ and optionally an auxiliary cross-entropy on the raw logits. Learn or tune $A$ using validation data, constraining rows to the simplex; start with a persistence parameter $a\in[0.8,0.99]$. 3. Computed from the paper: the Bayesian recursion and bursty missingness matrix. Estimated empirically: $A$, $a$, the dropout rate $\rho$, burst length $b$, and neural emissions. Estimate $\rho$ and $b$ directly from mask run lengths, and select $a$ by validation negative log-likelihood or by matching the empirical label dwell time. 4. First cheap experiment: train a small GRU or transformer classifier on sequential MNIST with synthetic three-class latent regimes, and separately on a real event sequence dataset such as Human Activity Recognition. Compare raw per-frame softmax, ordinary exponential moving average, and this Bayesian wrapper under clean, i.i.d., and bursty masking. Use $K=3$, $\rho=0.3$, and burst lengths $b\in\{2,5,10\}$. Measure accuracy, negative log-likelihood, expected calibration error, and flip rate. The quantitative prediction is that perturbations during missing runs decay according to $|\lambda_A|^q$ and that flip rate decreases monotonically with $a$ or $b$, while responsiveness worsens when the effective memory $\tau\approx b$ becomes longer than the event duration. Sweep $a$ and verify that the observed belief autocorrelation time agrees with $-1/\log|\lambda_A|$ within 20%; also test whether burst-trained marginalization loses substantially less accuracy than sentinel imputation as $\rho$ increases.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built and numerically verified the Bayesian logit smoother with bursty-mask marginalization. Exact class-belief contraction matched |lambda_A|^q with maximum error 2.2e-16; measured dropped-burst means matched targets 2, 5, and 10 within 2.1%; neutral missing updates had zero error. In the synthetic test, the method substantially beat sentinel imputation under bursty masking in NLL (0.499 vs 1.304) and calibration, while accuracy was similar to raw predictions for long bursts.

### Mechanism check

- Verdict: Built and numerically verified the Bayesian logit smoother with bursty-mask marginalization. Exact class-belief contraction matched |lambda_A|^q with maximum error 2.2e-16; measured dropped-burst means matched targets 2, 5, and 10 within 2.1%; neutral missing updates had zero error. In the synthetic test, the method substantially beat sentinel imputation under bursty masking in NLL (0.499 vs 1.304) and calibration, while accuracy was similar to raw predictions for long bursts.
- Confidence: 9/10
- Limitations: Only a synthetic HMM with simulated neural logits was tested. No trained neural classifier, real dataset, learned transitions, multi-seed confidence intervals, or FLOP/runtime study was included.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/1012/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/1012/bench_report.json)
- [bursty_latent_sequence.py](https://synthcore.org/code/1012/bursty_latent_sequence.py)
- [experiment.py](https://synthcore.org/code/1012/experiment.py)
- [report.md](https://synthcore.org/code/1012/report.md)
- [report_bench_2026-09-01T182433.md](https://synthcore.org/code/1012/report_bench_2026-09-01T182433.md)
- [results.json](https://synthcore.org/code/1012/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1012)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.