# Completely Monotone Multiscale Attention Decay

- ID: 268
- Canonical URL: https://synthcore.org/idea/268/completely-monotone-multiscale-attention-decay
- API JSON: https://synthcore.org/api/idea/268.json
- API Markdown: https://synthcore.org/api/idea/268.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.04467](https://arxiv.org/abs/2607.04467)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: attention, ssm, transformer, training-dynamics
- Math tags: probability, measure-theory, approximation-theory, functional-analysis
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Parameterize a relative-position or lag-decay function as a finite positive mixture of exponentials instead of learning arbitrary attention bias values. The resulting kernel is completely monotone on positive distances, so it is nonnegative, decreasing, and has alternating derivative signs; the mixture provides several learned memory scales without allowing oscillatory or unstable long-range biases.

## Mathematical statement

Proposition 2.2(i) states that a Bernstein function has the Levy-Khintchine representation $g(x)=a+bx+\int_{(0,\infty)}(1-e^{-xt})\,\nu(dt)$, where $a,b\geq 0$ and $\nu$ is a measure satisfying $\int(1\wedge t)\,\nu(dt)<\infty$. Proposition 2.2(ii) states that if $\psi$ is a special Bernstein function with $\psi(0)=0$, then its reciprocal has the potential representation $1/\psi(\lambda)=b+\int_0^\infty e^{-\lambda t}u(t)\,dt$, where $b\geq0$ and $u(t)$ has a nonincreasing locally integrable version. The transferable object is the Laplace mixture $K(d)=\int_0^\infty e^{-td}\,\mu(dt)$ with a nonnegative measure $\mu$: for every integer $n\geq0$, $(-1)^nK^{(n)}(d)=\int t^n e^{-td}\,\mu(dt)\geq0$. Use the finite approximation $K(d)=\sum_{j=1}^m w_j e^{-\tau_j d}$ with $w_j\geq0$ and $\tau_j>0$; then $K(d)\geq0$ and $K'(d)=-\sum_jw_j\tau_j e^{-\tau_jd}\leq0$. Here $d$ is a nonnegative relative position or state lag, $w_j$ are mixture weights, and $\tau_j$ are decay rates.

## Key formulas

- $$g(x)=a+bx+\int_{(0,\infty)}(1-e^{-xt})\,\nu(dt),\qquad a,b\geq0,\quad \int_{(0,\infty)}(1\wedge t)\,\nu(dt)<\infty.$$
- $$\frac{1}{\psi(\lambda)}=b+\int_{0}^{\infty}e^{-\lambda t}u(t)\,dt,\qquad \lambda>0,$$
- $$K(d)=\sum_{j=1}^{m}w_j e^{-\tau_j d},\qquad w_j=\frac{e^{\alpha_j}}{\sum_{k=1}^{m}e^{\alpha_k}},\quad \tau_j=e^{\beta_j},$$
- $$(-1)^nK^{(n)}(d)=\sum_{j=1}^{m}w_j\tau_j^n e^{-\tau_jd}\geq0,\qquad n=0,1,2,\ldots$$

## Implementation notes

Integrate the construction into the relative-position bias or multiplicative decay of a causal Transformer attention head. For token positions i and j with lag d=i-j\geq0, retain the usual content logit q_i^    op k_j/\sqrt{d_k}, but add a completely monotone bias b(d)=\log(\epsilon+K(d)) or multiply the attention numerator by K(d). Use m=4--16 components, unconstrained parameters alpha[1:m] and beta[1:m], normalized weights w=softmax(alpha), and positive rates tau=softplus(beta)+tau_min. A concrete forward pass is: (1) compute d=i-j; (2) compute E[j]=exp(-d*tau[j]) for every lag and component; (3) compute K=E @ w; (4) set logits=qk/sqrt(d_k)+gamma*log(K+epsilon), with learned scalar gamma constrained nonnegative by softplus, or use logits=qk/sqrt(d_k)+log(K+epsilon) directly; (5) apply the standard causal softmax. Initialize tau on a log-uniform grid covering short to long contexts, alpha uniformly, and gamma around 1. The paper supplies the positivity and alternating-derivative guarantees; the engineer only estimates or tunes tau, w, and gamma by backpropagation. Empirically monitor K(d), its derivative, attention entropy, gradient norms, and the fraction of heads concentrating on pathological long-range or short-range lags. First test on a 2-layer small Transformer for WikiText-2 or synthetic copy and retrieval tasks, comparing learned arbitrary relative biases, ALiBi, and RoPE at equal parameter count. Success means lower validation perplexity or better length-generalization on 2x training context, with fewer attention-logit explosions and no increase in training instability. An efficient implementation precomputes the m-by-context matrix E once per sequence length, costing O(mL) rather than adding pairwise learned parameters; for SSMs, the identical K(d) can replace a single exponential state decay with a positive mixture of m decay modes.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a reproducible finite positive exponential-mixture decay module and toy length-extrapolation experiment. The numerical check showed K(d)>0, strictly decreasing finite differences, and positive alternating-sign derivative expressions through order 4. On a synthetic two-scale target trained at length 32 and evaluated at length 64, the mixture achieved KL 6.6e-7 versus 0.01245 for a learned finite lag table with boundary extrapolation, indicating the proposed smooth multiscale parameterization extrapolates as intended. This is a promising toy signal, not evidence of improved language-model accuracy.

### Mechanism check

- Verdict: Built a reproducible finite positive exponential-mixture decay module and toy length-extrapolation experiment. The numerical check showed K(d)>0, strictly decreasing finite differences, and positive alternating-sign derivative expressions through order 4. On a synthetic two-scale target trained at length 32 and evaluated at length 64, the mixture achieved KL 6.6e-7 versus 0.01245 for a learned finite lag table with boundary extrapolation, indicating the proposed smooth multiscale parameterization extrapolates as intended. This is a promising toy signal, not evidence of improved language-model accuracy.
- Confidence: 7/10
- Limitations: Only a synthetic attention-distribution fitting task was tested, not a Transformer, WikiText, copy/retrieval task, ALiBi, RoPE, FLOPs, runtime, entropy, gradient stability, or learned content-dependent attention. The baseline extrapolation used repeated oldest-lag bias, and the mixture had a higher training cross-entropy despite much better length extrapolation; broader seeds and optimization comparisons were not performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.5058
- Paired wins: 3/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/1246/bench_report.json)
- [experiment.py](https://synthcore.org/code/1246/experiment.py)
- [report.md](https://synthcore.org/code/1246/report.md)
- [report_bench_2026-09-03T221133.md](https://synthcore.org/code/1246/report_bench_2026-09-03T221133.md)
- [results.json](https://synthcore.org/code/1246/results.json)
- [stage2_bench.py](https://synthcore.org/code/1246/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1246)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.