△ Mechanism confirmed, baseline not beaten 2026

Completely Monotone Multiscale Attention Decay

Implementation & benchmark of arXiv:2607.04467 — Bernstein Functions at Work: Coalescents, Copulas, and Subordination

Usefulness6/10

Difficulty4/10

Novelty7/10

Architecture Stability Accuracy Generalization Attention Ssm Transformer Training dynamics Probability Measure theory Approximation theory Functional analysis

Source paper: Bernstein Functions at Work: Coalescents, Copulas, and Subordination arXiv:2607.04467 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Parameterize a relative-position or lag-decay function as a finite positive mixture of exponentials instead of learning arbitrary attention bias values. The resulting kernel is completely monotone on positive distances, so it is nonnegative, decreasing, and has alternating derivative signs; the mixture provides several learned memory scales without allowing oscillatory or unstable long-range biases.

Formulas

$$g(x)=a+bx+\int_{(0,\infty)}(1-e^{-xt})\,\nu(dt),\qquad a,b\geq0,\quad \int_{(0,\infty)}(1\wedge t)\,\nu(dt)<\infty.$$

$$\frac{1}{\psi(\lambda)}=b+\int_{0}^{\infty}e^{-\lambda t}u(t)\,dt,\qquad \lambda>0,$$

$$K(d)=\sum_{j=1}^{m}w_j e^{-\tau_j d},\qquad w_j=\frac{e^{\alpha_j}}{\sum_{k=1}^{m}e^{\alpha_k}},\quad \tau_j=e^{\beta_j},$$

$$(-1)^nK^{(n)}(d)=\sum_{j=1}^{m}w_j\tau_j^n e^{-\tau_jd}\geq0,\qquad n=0,1,2,\ldots$$

Mathematical statement

Proposition 2.2(i) states that a Bernstein function has the Levy-Khintchine representation $g(x)=a+bx+\int_{(0,\infty)}(1-e^{-xt})\,\nu(dt)$, where $a,b\geq 0$ and $\nu$ is a measure satisfying $\int(1\wedge t)\,\nu(dt)<\infty$. Proposition 2.2(ii) states that if $\psi$ is a special Bernstein function with $\psi(0)=0$, then its reciprocal has the potential representation $1/\psi(\lambda)=b+\int_0^\infty e^{-\lambda t}u(t)\,dt$, where $b\geq0$ and $u(t)$ has a nonincreasing locally integrable version. The transferable object is the Laplace mixture $K(d)=\int_0^\infty e^{-td}\,\mu(dt)$ with a nonnegative measure $\mu$: for every integer $n\geq0$, $(-1)^nK^{(n)}(d)=\int t^n e^{-td}\,\mu(dt)\geq0$. Use the finite approximation $K(d)=\sum_{j=1}^m w_j e^{-\tau_j d}$ with $w_j\geq0$ and $\tau_j>0$; then $K(d)\geq0$ and $K'(d)=-\sum_jw_j\tau_j e^{-\tau_jd}\leq0$. Here $d$ is a nonnegative relative position or state lag, $w_j$ are mixture weights, and $\tau_j$ are decay rates.

Implementation notes

Integrate the construction into the relative-position bias or multiplicative decay of a causal Transformer attention head. For token positions i and j with lag d=i-j\geq0, retain the usual content logit q_i^ op k_j/\sqrt{d_k}, but add a completely monotone bias b(d)=\log(\epsilon+K(d)) or multiply the attention numerator by K(d). Use m=4--16 components, unconstrained parameters alpha[1:m] and beta[1:m], normalized weights w=softmax(alpha), and positive rates tau=softplus(beta)+tau_min. A concrete forward pass is: (1) compute d=i-j; (2) compute E[j]=exp(-d*tau[j]) for every lag and component; (3) compute K=E @ w; (4) set logits=qk/sqrt(d_k)+gamma*log(K+epsilon), with learned scalar gamma constrained nonnegative by softplus, or use logits=qk/sqrt(d_k)+log(K+epsilon) directly; (5) apply the standard causal softmax. Initialize tau on a log-uniform grid covering short to long contexts, alpha uniformly, and gamma around 1. The paper supplies the positivity and alternating-derivative guarantees; the engineer only estimates or tunes tau, w, and gamma by backpropagation. Empirically monitor K(d), its derivative, attention entropy, gradient norms, and the fraction of heads concentrating on pathological long-range or short-range lags. First test on a 2-layer small Transformer for WikiText-2 or synthetic copy and retrieval tasks, comparing learned arbitrary relative biases, ALiBi, and RoPE at equal parameter count. Success means lower validation perplexity or better length-generalization on 2x training context, with fewer attention-logit explosions and no increase in training instability. An efficient implementation precomputes the m-by-context matrix E once per sequence length, costing O(mL) rather than adding pairwise learned parameters; for SSMs, the identical K(d) can replace a single exponential state decay with a positive mixture of m decay modes.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built a reproducible finite positive exponential-mixture decay module and toy length-extrapolation experiment. The numerical check showed K(d)>0, strictly decreasing finite differences, and positive alternating-sign derivative expressions through order 4. On a synthetic two-scale target trained at length 32 and evaluated at length 64, the mixture achieved KL 6.6e-7 versus 0.01245 for a learned finite lag table with boundary extrapolation, indicating the proposed smooth multiscale parameterization extrapolates as intended. This is a promising toy signal, not evidence of improved language-model accuracy.

Agent confidence: 7/10

Baseline: Finite learned lag table: test KL 0.0124517934; train cross-entropy 2.4541333
Idea: 4-component positive exponential mixture: test KL 0.0000006586; train cross-entropy 3.3014403; K(d) minimum 0.0904 and maximum finite-difference dK -0.0002452

Limitations: Only a synthetic attention-distribution fitting task was tested, not a Transformer, WikiText, copy/retrieval task, ALiBi, RoPE, FLOPs, runtime, entropy, gradient stability, or learned content-dependent attention. The baseline extrapolation used repeated oldest-lag bias, and the mixture had a higher training cross-entropy despite much better length extrapolation; broader seeds and optimization comparisons were not performed.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.04467) Sep 3, 2026

Ahead of the baseline (−0.0%) but not statistically significant (p=0.5058, wins 3 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.2124
Idea mean: 0.2124
Effect (Δ): -0 (−0.0%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.5058 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.0%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in sequence forecasting track was tested; no CIFAR, dynamics, language modeling, length extrapolation, ALiBi/RoPE comparison, runtime benchmark, or larger-context evaluation was performed. The benchmark used 800 training examples, 300 test examples, 10 epochs, and three learning rates; the baseline decisive bias-strength knob was not independently swept because the canonical baseline was the learned lag table itself and the shared gamma was fixed at 1.0.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 3, 2026 latest Failed ✗

Agent confidence: 9/10

Built a causal two-layer transformer on the structurally matched sequence forecasting track, comparing an unconstrained learned relative-lag table against an 8-component positive exponential-mixture bias with identical training conditions. The mechanism signature was confirmed from the trained models: maximum bias first difference was -0.04459, while total content-dependent attention was not monotone. However, baseline and idea test MSE were effectively identical (0.2124029454 vs 0.2124029398), with paired delta -5.6e-9 and permutation p=0.5058; therefore there is no observed accuracy win.

Baseline: test MSE mean 0.2124029453843832, std 0.029564272137155054; best lr 0.0015; per-seed [0.2271955609, 0.1741580069, 0.2089696825, 0.2423083037, 0.1920005530, 0.1792486161, 0.2660641670, 0.2092786729]
Idea: test MSE mean 0.21240293979644775, std 0.02956426589305894; best lr 0.0015; per-seed [0.2271955609, 0.1741579920, 0.2089696825, 0.2423083037, 0.1920005530, 0.1792486310, 0.2660641382, 0.2092786580]
Mechanism signature: map[confirmed:true n_models:8 observed_attention_max_first_difference:0.030105771829364122 observed_bias_max_first_difference:-0.044586181640625 observed_bias_min_second_difference:0.0005350112915039062 prediction:trained mixture log-kernel bias decreases with lag; retained content logits can make total attention nonmonotone]
Paired comparison: map[delta_mean:-5.587935447692871e-09 idea_wins:3 n_pairs:8 p_value:0.5058 verdict:no measurable effect]

Benchmark result