Multiplicative Adaptive Attention Graph
Implementation & benchmark of arXiv:2607.21110 — The mean-field limit of non-exchangeable particle systems with non-conservative dynamics and adaptive weights
Source paper: The mean-field limit of non-exchangeable particle systems with non-conservative dynamics and adaptive weights arXiv:2607.21110 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Give each query-token pair a positive adaptive edge weight that evolves by a multiplicative rule instead of relying only on instantaneous dot-product attention logits. Edges whose aggregate interaction is useful can grow, while overloaded or incompatible neighborhoods can shrink. Sparse initialization is preserved because an edge initialized at zero remains zero under the multiplicative dynamics.
Formulas
Mathematical statement
The paper studies the coupled system $$\partial_t X_i(t)=\sum_{k=1}^{N}w_{ik}(t)K_1(X_i(t)-X_k(t)),$$ and $$\partial_t w_{ji}(t)=w_{ji}(t)\left[A-\sum_{k=1}^{N}w_{ik}(t)K_2(X_i(t)-X_k(t))\right].$$ Here $X_i(t)\in\mathbb{R}^d$ is the state of particle or token $i$, $w_{ij}(t)\geq 0$ is a directed interaction weight, $A>0$ is a baseline growth parameter, and $K_1,K_2$ are deterministic interaction kernels. The multiplicative factor $w_{ji}(t)$ gives positivity preservation: positive weights remain positive, and zero weights remain zero. For a sparse attention neighborhood $\mathcal{N}(i)$, use the index-consistent adaptation $$\dot w_{ij}=w_{ij}\left[A-\sum_{k\in\mathcal{N}(i)}w_{ik}K_2(h_i-h_k)\right],$$ where $h_i$ is the current token representation. An exponential Euler discretization is $$w_{ij}^{t+1}=w_{ij}^{t}\exp\left(\eta_w\left[A-\sum_{k\in\mathcal{N}(i)}w_{ik}^{t}K_2(h_i^t-h_k^t)\right]\right).$$ Combine the adaptive weight with ordinary attention through $$\alpha_{ij}^{t}=\frac{w_{ij}^{t}\exp(q_i^\top k_j/\sqrt{d})}{\sum_{\ell\in\mathcal{N}(i)}w_{i\ell}^{t}\exp(q_i^\top k_\ell/\sqrt{d})}.$$ The vectors $q_i,k_j,v_j$ are query, key, and value vectors, $d$ is head dimension, and $\alpha_{ij}$ is used to aggregate values.
Implementation notes
(1) Integration point: modify a standard sparse self-attention layer. Maintain a positive edge state W with shape batch by heads by sequence length by neighborhood size. Each query position i has a fixed permitted-neighbor list N(i), such as a local window plus a few random or global edges. Store log W rather than W to avoid numerical underflow, and never allocate a dense sequence-length-by-sequence-length matrix. Initialize every permitted edge with w_ij=1 and omit forbidden edges entirely. (2) Pseudocode: compute Q, K, and V; gather neighboring keys, values, and detached token states; compute s_ij=q_i dot k_j divided by sqrt(d); compute r_ik=K2(h_i-h_k); compute g_i=A minus the sum over k in N(i) of w_ik times r_ik; update logw_ij by logw_ij plus eta_w times g_i; clamp logw to a conservative interval such as [-8,8]; compute alpha by applying a neighborhood softmax to s_ij plus logw_ij; return the residual value aggregation. Use K2(z)=softplus(u dot z+b), where u and b are learned parameters, or begin with K2(z)=||z|| squared divided by (2 sigma squared). Update W once per optimizer step using detached representations, optionally smoothing g_i with an exponential moving average. (3) The positivity and support-preservation properties come from the paper's multiplicative ODE. The discrete step size, clipping, kernel choice, and detachment are empirical engineering choices. Estimate the aggregate interaction using only the sparse neighborhood sum. (4) First experiment: train a six-layer, 256-dimensional Transformer on WikiText-103 or a comparably sized language-model corpus with sequence length 512. Compare ordinary fixed sparse attention, static learned edge biases, and adaptive W at identical neighbor counts and FLOPs. Measure validation perplexity, loss versus optimizer steps, wall-clock overhead, gradient norms, row entropy, and the fraction of attention rows dominated by one edge. Success means lower perplexity at fixed sparse FLOPs, equal perplexity with fewer neighbors, or improved stability without exploding weights or NaNs.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.