Stochastic-order monotone attention ratios
Implementation & benchmark of arXiv:2607.02511 — A Probabilistic Sign Rule for Quotients of Positive Series and Integral Transforms
Source paper: A Probabilistic Sign Rule for Quotients of Positive Series and Integral Transforms arXiv:2607.02511 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Build an attention or positive-mixture module whose output ratio at two control settings is provably monotone in an ordered index such as token distance, retrieval rank, or discretized uncertainty. Use normalized-positive-series identities to replace an unstable quotient derivative with a difference of expectations, and penalize violations of the resulting stochastic-order condition during training.
Formulas
Mathematical statement
The paper starts from the positive-series representation of the Kummer function, $M(a;c;x)=\sum_{n\geq 0}\frac{(a)_n}{(c)_n}\frac{x^n}{n!}$, and studies positive quotients such as $M(a;c+\delta;x)/M(a;c;x)$ for $\delta>0$. The general construction is: for positive summands $A_n(\theta)>0$, define $Z(\theta)=\sum_n A_n(\theta)$ and the induced probability law $p_\theta(n)=A_n(\theta)/Z(\theta)$. Writing the score as $s_\theta(n)=\partial_\theta\log A_n(\theta)$ gives the moment identity $\partial_\theta\log Z(\theta)=\mathbb{E}_{p_\theta}[s_\theta(N)]$. For two positive series $Z_1,Z_0$, the quotient rule becomes $\partial_\theta\log(Z_1/Z_0)=\mathbb{E}_{p_1}[s_1(N)]-\mathbb{E}_{p_0}[s_0(N)]$. In the shared-score case $s_1=s_0=s$, first-order stochastic dominance of $p_1$ over $p_0$ and monotonicity of $s(n)$ determine the sign of the quotient derivative. For implementation, $n$ is an ordered token or component index, $A_n$ is an attention contribution or positive mixture term, and the two laws correspond to two values of a scalar control.
Implementation notes
Integrate this into a transformer attention block or retrieval mixture, not into the value projection. Choose an ordered component index $n$: token distance, retrieval rank, uncertainty bin, or a fixed discretization of a scalar control. For each query, form positive contributions $A_n^{(r)}=\exp(\ell_n^{(r)})v_n$ at two control settings $r\in\{0,1\}$, where $v_n>0$ is a scalar gate or evidence magnitude and $\ell_n^{(r)}$ is the control-dependent logit. Compute $Z_r=\sum_n A_n^{(r)}$ and $p_r(n)=A_n^{(r)}/(Z_r+\varepsilon)$. The quotient score is $q=\log Z_1-\log Z_0$.
Pseudocode: for each batch: logits0, logits1 = module(x, control0), module(x, control1); A0=exp(logits0)*positive_gate; A1=exp(logits1)*positive_gate; p0=A0/(A0.sum(dim=n)+eps); p1=A1/(A1.sum(dim=n)+eps); score0=dlogA_dcontrol(control0); score1=dlogA_dcontrol(control1); qgrad=(p1*score1).sum(n)-(p0*score0).sum(n); violation=max(0, margin-sign_target*qgrad); loss += lambda*violation. Estimate the scores analytically from the control-dependent logit parameterization, or with the finite difference $(\log A(\theta+h)-\log A(\theta-h))/(2h)$. Estimate first-order stochastic dominance with cumulative probabilities $F_r(k)=\sum_{n\leq k}p_r(n)$; for the desired direction, penalize $\sum_k\max(0,F_1(k)-F_0(k))$. The normalized-series identities are exact for finite positive sums; stochastic dominance and score monotonicity are empirical constraints. First test on a small two-layer transformer and a synthetic task whose target is known to increase with distance reliability, then on a small retrieval-QA benchmark. Compare ordinary attention and the regularized module at equal FLOPs. Success means fewer control-response inversions, smoother outputs under perturbations, and equal or improved validation accuracy; additionally verify on held-out controls that the quotient derivative has the predicted sign.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.