# GKP Log-Concave Lag Mixer

- ID: 260
- Canonical URL: https://synthcore.org/idea/260/gkp-log-concave-lag-mixer
- API JSON: https://synthcore.org/api/idea/260.json
- API Markdown: https://synthcore.org/api/idea/260.md
- Verification status: unverified
- Source: [arXiv:2607.04217](https://arxiv.org/abs/2607.04217)
- Category: architecture
- Solves: stability, generalization, memory
- ML areas: attention, ssm, rnn, convolution
- Math tags: combinatorics, inequalities, algebra, total-positivity
- Ratings: usefulness 5/10; difficulty 3/10; novelty 6/10

## Idea description

Generate temporal attention or convolution weights with the Graham–Knuth–Patashnik recurrence instead of learning every lag weight independently. For nonnegative recurrence parameters, the resulting lag sequence is strongly log-concave, so its normalized kernel is naturally unimodal and suppresses high-frequency sign-free oscillations without requiring a separate smoothness penalty. The six parameters can be learned per head, channel group, or layer, giving O(1) learned parameters for an O(K) receptive field.

## Mathematical statement

The GKP array is defined by T(0,k)=delta_{k,0}, T(n,k)=0 for k<0 or k>n, and T(n,k)=(alpha n+beta k+gamma)T(n-1,k)+(alpha' n+beta' k+gamma')T(n-1,k-1). Here n is the recurrence depth, k is the lag or scale index, and alpha,beta,gamma,alpha',beta',gamma' are nonnegative parameters. The paper proves coefficientwise strong log-concavity in each fixed row: T(n,k)^2-T(n,k-1)T(n,k+1) has nonnegative coefficients in the six parameters; more generally, the cross-products satisfy T(n,k)T(n,l) >= T(n,k-1)T(n,l+1) for k <= l. The symbol >= here denotes coefficientwise comparison of polynomials. For numerical nonnegative parameter values this implies ordinary log-concavity, so ratios T(n,k+1)/T(n,k) are nonincreasing wherever defined and the row has no separated secondary peaks. Use the row as a positive lag kernel after normalization.

## Key formulas

- $$T(n,k)=(\alpha n+\beta k+\gamma)T(n-1,k)+(\alpha^{\prime}n+\beta^{\prime}k+\gamma^{\prime})T(n-1,k-1),\qquad T(0,k)=\delta_{k,0},\quad T(n,k)=0\ \text{if }k<0\text{ or }k>n.$$
- $$T(n,k)^2-T(n,k-1)T(n,k+1)\succeq 0,$$
- $$T(n,k)T(n,\ell)\succeq T(n,k-1)T(n,\ell+1)\qquad (k\leq \ell),$$
- $$w_k=\frac{T(K,k;\bm\mu)+\varepsilon}{\sum_{j=0}^{K}(T(K,j;\bm\mu)+\varepsilon)},\qquad y_t=\sum_{k=0}^{K}w_kV_{t-k}.$$

## Implementation notes

Integrate this at the lag-weight generation point of a causal temporal mixer, local attention head, or depthwise 1-D convolution. Let K be the maximum lag and learn six unconstrained scalars per head or channel group. Convert them to nonnegative recurrence parameters with softplus: alpha=softplus(a), beta=softplus(b), gamma=softplus(c), alpha_p=softplus(a_p), beta_p=softplus(b_p), gamma_p=softplus(c_p). At every forward pass compute a row of length K+1 using the recurrence below; use n=K, or optionally concatenate several rows n=1,...,K and learn a softmax over rows. Pseudocode: `T=zeros(K+1); T[0]=1; for n in 1..K: new=zeros(K+1); for k in 0..n: new[k]=(alpha*n+beta*k+gamma)*T[k] + (alpha_p*n+beta_p*k+gamma_p)*(T[k-1] if k>0 else 0); T=new; w=(T+eps)/(sum(T)+eps*(K+1)); y=sum_k w[k]*V[t-k]`. Compute the recurrence in float32 even under mixed precision, then cast w to the activation dtype. The mathematical guarantee is exact only for nonnegative fixed parameters; do not add unconstrained per-lag perturbations if preserving log-concavity is the goal. Empirically record the number of sign changes in first differences of w and the fraction of rows violating `w[k]^2 >= w[k-1]*w[k+1]`. First experiment: compare this mixer against a learned softmax lag kernel, depthwise convolution, and ALiBi-style local attention in a 6-layer small Transformer on Copy, Adding, and WikiText-2 sequence-length extrapolation tasks. Match parameter count and multiply-adds. Success means lower validation perplexity or copy error at equal compute, improved extrapolation to 2x training context, and substantially fewer multimodal lag kernels; a secondary test is whether the structured mixer remains stable when K is increased without retraining all lag parameters.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.