Finite-Population Binomial Rate Layer

Integrate this into a recurrent state-space block, an MoE router, or a sparse population MLP at the point where a continuous activation fraction is produced. Let logits \(z_{t,\mu}\) produce a rate \(r_{t,\mu}=\mathcal F_\mu(\mathbf m_{t-1})\) using sigmoid or softmax; choose an effective population size \(N_\mu\) and bin width \(T\) so that \(0\leq Tr_{t,\mu}\leq1\). During training, sample \(K_{t,\mu}\sim\operatorname{Binomial}(N_\mu,Tr_{t,\mu})\), set \(m_{t,\mu}=K_{t,\mu}/(N_\mu T)\), and feed \(m_t\) to the next block. Use a straight-through estimator for the sampled count, or use the reparameterized approximation \(m_t=r_t+\sqrt{r_t(1-Tr_t)/(N_\mu T)}\,\epsilon_t\) with clipped Gaussian \(\epsilon_t\) for automatic differentiation. At inference, use the exact mean \(m_t=r_t\) or sample only when uncertainty is desired. The paper supplies the mean and variance; the engineer estimates \(N_\mu\), \(T\), and the rate parameterization empirically. First test on a small GRU or 2-layer Transformer MoE router on CIFAR-10 or WikiText-2, comparing deterministic routing and standard dropout at equal FLOPs. Measure validation accuracy or perplexity, gradient variance, and robustness to batch-size reduction. Success is lower validation loss or better corruption robustness at matched parameter count, with variance scaling approximately as \(1/N_\mu\).