{
 "artifacts": [
  {
   "name": "gated_attention.py",
   "url": "https://synthcore.org/code/1041/gated_attention.py"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/1041/report.md"
  },
  {
   "name": "results.json",
   "url": "https://synthcore.org/code/1041/results.json"
  },
  {
   "name": "run_experiment.py",
   "url": "https://synthcore.org/code/1041/run_experiment.py"
  },
  {
   "name": "stage2_bench.py",
   "url": "https://synthcore.org/code/1041/stage2_bench.py"
  }
 ],
 "category": "architecture",
 "description": "Replace dense graph self-attention with two parallel branches: exact softmax attention only over graph neighbors and a global linear-attention branch that summarizes all nodes through feature-space statistics. A learned node-wise gate interpolates between the branches, allowing locally structured nodes to use sparse attention while retaining a global-information path.",
 "download_zip": "https://synthcore.org/download/1041",
 "formulas_latex": [
  "$$L_i=\\sum_{j:A_{ij}=1}\\frac{\\exp(q_i k_j^\\top/\\sqrt r)}{\\sum_{\\ell:A_{i\\ell}=1}\\exp(q_i k_\\ell^\\top/\\sqrt r)}v_j,\\qquad q_i=x_iW_Q,\\quad k_j=x_jW_K,\\quad v_j=x_jW_V.$$",
  "$$G_i=\\frac{\\phi(q_i)^\\top S}{\\phi(q_i)^\\top z+\\epsilon},\\qquad S=\\sum_{j=1}^{N}\\phi(k_j)v_j^\\top,\\qquad z=\\sum_{j=1}^{N}\\phi(k_j),$$",
  "$$g_i=\\sigma(x_iw_g+b_g),\\qquad Y_i=g_iL_i+(1-g_i)G_i.$$",
  "$$T_{\\mathrm{hybrid}}=O(|E|r+Nr^2),\\qquad T_{\\mathrm{dense}}=O(N^2r),\\qquad \\mathrm{speedup}\\approx\\frac{N^2r}{|E|r+Nr^2}.$$"
 ],
 "id": 2699,
 "implementation": "1. Integration point: implement this as a replacement for dense self-attention in a graph transformer or spatiotemporal graph model. Inputs are node states `X [batch,N,d]` and a sparse adjacency list or mask `A`; retain the usual residual connection, normalization, multi-head splitting, and feed-forward block. 2. Pseudocode: compute `Q,K,V`; for each head, gather only edge pairs `(i,j)` and apply segmented softmax to obtain `L`; compute `PhiQ=elu(Q)+1` and `PhiK=elu(K)+1`; accumulate `S=sum_j outer(PhiK[j],V[j])` and `z=sum_j PhiK[j]`; form `G=(PhiQ @ S)/(PhiQ @ z + eps)`; compute `g=sigmoid(linear(X))`; return `LayerNorm(X + Wo(g*L+(1-g)*G))`. Use sparse GPU kernels or packed edge lists rather than materializing an $N\\times N$ mask. 3. Computed directly from the mechanism: the adjacency-constrained local attention, global linear statistics, and node-wise gate. Estimated empirically: the best kernel map, gate regularization, sparse-kernel overhead, and accuracy loss. 4. First cheap experiment: compare this layer against dense graph attention on Cora or ogbn-arxiv, plus a synthetic task whose labels depend separately on one-hop and global graph statistics. Sweep $N$ and average degree while matching hidden width and parameter count. Measure wall-clock latency, peak memory, accuracy, and gate values. The predicted signature is quadratic dense runtime versus approximately $|E|r+Nr^2$ hybrid runtime; for bounded-degree graphs, runtime should become approximately linear in $N$. Once sparse-kernel overhead is amortized, the hybrid should achieve at least a 2x speedup, while retaining accuracy within 1-2 percentage points. On the synthetic mixed task, local-only nodes should have $g_i$ near one and global-only nodes should have $g_i$ near zero.",
 "math_summary": "Let $X\\in\\mathbb{R}^{N\\times d}$ be node features, $A\\in\\{0,1\\}^{N\\times N}$ the fixed graph adjacency including self-loops, and $Q=XW_Q$, $K=XW_K$, $V=XW_V$ with $W_Q,W_K,W_V\\in\\mathbb{R}^{d\\times r}$. The local branch computes masked softmax attention over neighbors, so its cost depends on the edge count $|E|$ rather than $N^2$. The global branch uses kernelized linear attention with a nonnegative feature map $\\phi$. A node-wise gate $g_i$ combines the two outputs. The resulting computational cost is $O(|E|r+Nr^2)$ instead of dense attention cost $O(N^2r)$, assuming the feature dimension $r$ is smaller than $N$.",
 "math_tags": [
  "linear-algebra",
  "graph-theory",
  "optimization"
 ],
 "ml_areas": [
  "graph-nn",
  "attention",
  "transformer",
  "inference-speedup"
 ],
 "paper": {
  "arxiv_id": "2608.23414",
  "arxiv_url": "https://arxiv.org/abs/2608.23414",
  "summary_what_math_gives_to_ml": "The paper offers a concrete hybrid attention mechanism for graph-structured systems: restrict one attention branch to physically or communicationally valid neighbors, add a global linear-mixing branch, and combine them with a node-wise learnable gate. Its transferable asset is a computational decomposition that preserves local relational selectivity while avoiding the quadratic cost of dense global attention. The direct neural-network transfer is a drop-in GraphGHHA layer for graph transformers, spatiotemporal forecasting, molecular networks, and multi-agent models, with the adjacency mask supplied by the task graph. The key falsifiable prediction is that attention cost scales with the number of edges plus a linear global term, rather than with the square of the node count.",
  "title": "Graph-Native Attention Acceleration for Attack Detection in Cyber-Physical Systems",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 5,
  "usefulness": 7
 },
 "solves": [
  "speedup",
  "scalability",
  "accuracy"
 ],
 "title": "Gated Local-Global Graph Attention",
 "url": "https://synthcore.org/idea/2699/gated-local-global-graph-attention",
 "verification": {
  "peer_reviewed": false,
  "stage1_mechanism_check": {
   "worked": true,
   "confidence": 8,
   "verdict": "Built a PyTorch gated local-global attention MVP with edge-list local softmax, ELU+1 linear attention, and a learned node gate. The mathematical checks confirmed masked local attention and linear-kernel reassociation to below 2.4e-7 error, while the operation proxy ratio grew from 2.67x at N=64 to 21.33x at N=512 with observed log-log slope 1.00, matching the predicted linear-in-N scaling for fixed degree. The hybrid reached lower synthetic mixed-task MSE than the dense baseline (2.26e-5 vs 6.53e-2), but it was slower in this small unoptimized implementation at every tested size, so the speedup claim was not observed.",
   "metrics": {
    "baseline": "Dense attention: synthetic MSE 0.065324; latency N=64/128/256/512: 0.328/0.619/0.502/0.345 ms",
    "idea": "Hybrid attention: synthetic MSE 0.0000226; latency N=64/128/256/512: 1.934/1.809/1.460/1.956 ms; operation-proxy ratio dense/hybrid grows 2.67x to 21.33x"
   },
   "how_to_run": "python3 run_experiment.py",
   "files": [
    "gated_attention.py",
    "run_experiment.py",
    "results.json"
   ],
   "limitations": "Only a single-head toy synthetic task was tested; no Cora, ogbn-arxiv, multi-head transformer block, validation split, gate-value specialization analysis, backward-pass benchmark, memory measurement, or optimized sparse GPU kernel was included. Timing was dominated by Python/PyTorch edge-list overhead and does not validate production-scale speedups."
  },
  "status": "unverified",
  "status_label": "Unverified",
  "updated_at": "2026-09-01T20:45:15",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}