Unverified 2026

Recursive temporal attention spanner

Implementation & benchmark of arXiv:2609.02851 — Almost Linear 3-Spanners of Temporal Cliques

Usefulness6/10
Difficulty6/10
Novelty6/10

Source paper: Almost Linear 3-Spanners of Temporal Cliques arXiv:2609.02851 · analyzed Sep 3, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Construct a sparse directed attention mask over ordered tokens by treating token pairs as edges labeled by time or position. Apply a recursive three-hop spanner construction so that every dense temporal route has a sparse route using at most three times as many hops. Run attention only on the selected edges and use three sparse message-passing layers, or a small learned hop-composition block, to approximate dense causal attention.

Formulas

$$\lambda' (\{u_S,v_T\})=\begin{cases}\lambda(\{u,v\}),&u\neq v,\\1,&u=v,\end{cases}$$
$$K(n)\leq 2n+2K(f(n))+K(n-f(n)),\qquad n\geq 3,$$
$$K(n)\leq n^2,\qquad n\geq 1,$$
$$\operatorname{Attn}_{\mathrm{sparse}}(Q,K,V)_i=\sum_{j\in N_{\mathrm{span}}(i)}\frac{\exp(q_i^\top k_j/\sqrt d)}{\sum_{\ell\in N_{\mathrm{span}}(i)}\exp(q_i^\top k_\ell/\sqrt d)}v_j$$

Mathematical statement

A temporal graph is G=(V,E,lambda), where lambda(e) is a positive integer timestamp and a temporal path is a path whose edge labels are nondecreasing. A temporal alpha-spanner preserves every pair's minimum-hop temporal distance up to multiplicative factor alpha. The paper reduces a temporal clique to a balanced temporal biclique: S={u_S:u in V} and T={u_T:u in V}, with lambda'({u_S,v_T})=lambda({u,v}) for u != v and lambda'({u_S,u_T})=1. A temporal alpha-bispanner of this biclique maps back to a temporal alpha-spanner of the original clique by deleting artificial same-vertex edges and compressing each edge {x_S,x_T} to the single vertex x. The key recursive guarantee is Lemma 6: for any choice f(n) in {1,...,n-1}, RecSpan_f returns a 3-bispanner and its worst-case size K(n) satisfies K(n) <= 2n + 2K(f(n)) + K(n-f(n)) for n >= 3, with the trivial bound K(n) <= n^2. Here n is the number of vertices on each side of the balanced biclique, f(n) is the recursively selected block size, and K is the maximum number of retained edges. In the neural adaptation, V is the token set, lambda is a discretized causal time or position label, and a retained biclique edge is an allowed attention interaction. The factor-three property means a temporally valid dense route of h interactions has a sparse route of at most 3h interactions, subject to the same nondecreasing-label constraint.

Implementation notes

Integrate this at the attention-logit and mask stage of a causal Transformer. For a sequence of n tokens, create two copies S and T of the token indices. Set the biclique edge label to the token's causal time difference or to a discretized event timestamp; set the diagonal S-T label to 1 as in the paper. Build a sparse edge set recursively. At a subproblem containing m source indices and m target indices, choose a block size f(m), initially f(m)=max(1,ceil(m*exp(-sqrt(log(max(m,3)))))). Add a linear-size routing skeleton using a greedy temporal 3-hop cover: repeatedly add the candidate edge that covers the largest number of currently uncovered ordered pairs (s,t) having a nondecreasing-label path of at most three edges, then stop after 2m edges or when all pairs are covered. For every candidate path, explicitly check label order. Recursively process the selected block twice and the complement once, as prescribed by the recurrence. Convert retained S-T edges to an n-by-n boolean attention mask, and use it in every layer or cycle masks across three consecutive layers so information can traverse certified paths. The paper supplies the target recurrence and the 3-hop invariant; the greedy cover and f schedule are engineering approximations and should be measured against exact dense-pair coverage. First run on WikiText-103 or synthetic timestamped sequences with n in {256,512,1024}, comparing dense causal attention, local attention, and random sparse attention at equal hidden size. Measure retained-edge count, GPU memory, wall-clock attention FLOPs, perplexity, and the fraction of source-target pairs whose minimum temporal hop distance increases by more than 3x. Success is a 2x or larger attention-memory reduction with less than 1% perplexity degradation and empirical 3-hop coverage above 99%.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.