# Spiderweb Hierarchical Attention

- ID: 1706
- Canonical URL: https://synthcore.org/idea/1706/spiderweb-hierarchical-attention
- API JSON: https://synthcore.org/api/idea/1706.json
- API Markdown: https://synthcore.org/api/idea/1706.md
- Verification status: beats_baseline
- Source: [arXiv:2608.02369](https://arxiv.org/abs/2608.02369)
- Category: architecture
- Solves: speedup, memory, scalability
- ML areas: attention, transformer, inference-speedup, memory
- Math tags: geometry, metric-geometry, graph-theory, hyperbolic-geometry
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace dense token-to-token attention by a multiscale spiderweb communication pattern. Tokens first aggregate upward through a dyadic hierarchy, communicate horizontally only with a small number of cells at the appropriate height, and then receive information broadcast downward. Hyperbolic distance supplies a principled rule for choosing the height at which two tokens interact: nearby tokens interact at fine scales, while far-apart tokens interact through coarse representatives.

## Mathematical statement

The paper defines the hyperbolic-type half-space metric on S=N    imes\mathbb{R}_{+} by d_{S}((x,y),(x',y'))=C_{N}\operatorname{arcosh}(1+(d_{N}(x,x')^{2}+(y-y')^{2})/(2yy')), where x,x' are base-space positions, y,y'>0 are heights, d_N is the base metric, and C_N=(\log\eta)^{-1} is a normalization. It discretizes S with dyadic cells Q and a unique parent operation. For a path \gamma=(z_0,\ldots,z_m) in the spiderweb graph, its height is h(\gamma)=\max_{0\le j\le m}\mathcal{h}(z_j). Lemma 3.5 states that for endpoints z_Q,z_R and any path of height h, there are unique ancestors Q_h,R_h at level h satisfying Q\subset Q_h and R\subset R_h. The standard-geodesic construction uses ascent through parent edges, a uniformly bounded horizontal segment between same-level ancestors, and descent. The neural adaptation uses the same ancestor routing: every token is assigned to a leaf cell Q, pooled to ancestors Q_l, performs sparse horizontal attention between a bounded neighbor set at level l, and broadcasts back downward. The arcosh metric determines the preferred level l for a token pair, while the bounded horizontal degree controls compute.

## Key formulas

- $$d_{S}\bigl((x,y),(x',y')\bigr)=C_{N}\,\operatorname{arcosh}\!\left(1+\frac{d_{N}(x,x')^{2}+(y-y')^{2}}{2yy'}\right)$$
- $$h(\gamma)=\max_{0\leq j\leq m}\mathcal{h}(z_j)$$
- $$Q\subset Q_h,\qquad R\subset R_h,\qquad Q_h,R_h\in\mathscr{D}_h$$
- $$\ell^{*}(i,j)=\arg\min_{\ell\in\{0,\ldots,L\}}\left|d_{S}\bigl((x_i,y_\ell),(x_j,y_\ell)\bigr)-\tau_\ell\right|$$

## Implementation notes

Integrate this as a replacement for one or more dense self-attention blocks in a Transformer encoder. For a sequence of n tokens, create a balanced dyadic hierarchy with levels l=0,...,L, where level 0 contains individual tokens and level L contains one or a few global cells. Let Q_l(i) be token i's ancestor at level l, let y_l=y_0\eta^l be its height, and let x_i=i/n for a sequence or the normalized 2D coordinate for an image. At each level, compute cell summaries with learned pooling: s_{Q_l}=\operatorname{Pool}({h_i:Q_l(i)=Q_l}), and construct keys and values from these summaries. For each cell Q at level l, define a fixed-size horizontal neighbor set N_l(Q), obtained from nearby cells in the base coordinate; use attention only between Q and N_l(Q). Finally broadcast the updated summary to descendants through learned gated residuals.

Pseudocode: (1) assign every token i to leaf cell Q_0(i); (2) for l=1,...,L, set Q_l(i)=parent(Q_{l-1}(i)) and compute s_{Q_l}; (3) for every level l and cell Q, compute q_Q,k_R,v_R for R in N_l(Q), and set a_{QR}=softmax_R(q_Q^Tk_R/\sqrt d+b_{QR}); (4) update s'_Q=\sum_{R\in N_l(Q)}a_{QR}v_R; (5) propagate s'_Q to all descendant tokens and combine with the local representation using h_i\leftarrow h_i+g_l(i)s'_{Q_l(i)}. The paper's mathematics supplies the parent hierarchy, height-based routing, and hyperbolic distance. Estimate the base distance d_N from token-coordinate distance; estimate or tune \eta, y_0, neighbor radius, and gates empirically. Initially use all levels, then prune levels whose attention mass is negligible.

The first experiment should use a 2K- or 4K-token language-modeling Transformer on WikiText-103 or a synthetic long-range-copy task. Compare dense attention, local-window attention with the same hidden size, and Spiderweb attention at matched parameter count. Measure validation perplexity at equal training FLOPs, wall-clock throughput, peak activation memory, and performance on dependencies longer than the local window. A successful result is near-dense perplexity with O(n\log n) rather than O(n^2) attention work, at least 1.5x throughput or substantially lower memory, and improved accuracy over a purely local baseline on long-range-copy examples. An ablation must compare learned routing against routing based directly on the hyperbolic distance formula.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: helps
- Verdict: Built a readable NumPy Spiderweb hierarchy with dyadic pooling, bounded same-level neighborhoods, attention updates, and downward broadcast. The math checks confirmed all three targeted predictions: equal-height hyperbolic distance decreased monotonically across levels, the inverse-metric transition level matched the discrete argmin with 96% exact agreement and mean error 0.04 levels, and normalized work remained bounded while dense work grew by 29.1x over the tested range. The impulse test showed global communication (far-token influence 0.00638 versus zero for local attention), while estimated work was 3,835 versus 65,536 dense pair scores; however, the unoptimized NumPy implementation was slower wall-clock than dense attention, so the demonstrated win is structural work/memory and long-range reach rather than runtime.

### Mechanism check

- Verdict: Built a readable NumPy Spiderweb hierarchy with dyadic pooling, bounded same-level neighborhoods, attention updates, and downward broadcast. The math checks confirmed all three targeted predictions: equal-height hyperbolic distance decreased monotonically across levels, the inverse-metric transition level matched the discrete argmin with 96% exact agreement and mean error 0.04 levels, and normalized work remained bounded while dense work grew by 29.1x over the tested range. The impulse test showed global communication (far-token influence 0.00638 versus zero for local attention), while estimated work was 3,835 versus 65,536 dense pair scores; however, the unoptimized NumPy implementation was slower wall-clock than dense attention, so the demonstrated win is structural work/memory and long-range reach rather than runtime.
- Confidence: 8/10
- Limitations: No learned Transformer, language-model perplexity, WikiText/long-range-copy training, matched parameter/FLOP optimization, GPU kernel, peak memory measurement, or hyperbolic-distance routing ablation was tested. The runtime comparison is not favorable because the reference implementation uses slow Python/NumPy loops and is not an optimized sparse kernel.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/630/bench_report.json)
- [bench_report_rerun.json](https://synthcore.org/code/630/bench_report_rerun.json)
- [bench_stage2.py](https://synthcore.org/code/630/bench_stage2.py)
- [benchmark_results.json](https://synthcore.org/code/630/benchmark_results.json)
- [impulse_check.py](https://synthcore.org/code/630/impulse_check.py)
- [impulse_results.json](https://synthcore.org/code/630/impulse_results.json)
- [report.md](https://synthcore.org/code/630/report.md)
- [report_bench_2026-08-31T203628.md](https://synthcore.org/code/630/report_bench_2026-08-31T203628.md)
- [run_benchmark.py](https://synthcore.org/code/630/run_benchmark.py)
- [spiderweb_experiment.py](https://synthcore.org/code/630/spiderweb_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/630)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.