✓✓ Beats tuned baseline 2026

Spiderweb Hierarchical Attention

Implementation & benchmark of arXiv:2608.02369 — Poincaré inequalities on hyperbolic-type spaces

Usefulness7/10

Difficulty5/10

Novelty6/10

Architecture Speedup Memory Scalability Attention Transformer Inference speedup Memory Geometry Metric geometry Graph theory Hyperbolic geometry

Source paper: Poincaré inequalities on hyperbolic-type spaces arXiv:2608.02369 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace dense token-to-token attention by a multiscale spiderweb communication pattern. Tokens first aggregate upward through a dyadic hierarchy, communicate horizontally only with a small number of cells at the appropriate height, and then receive information broadcast downward. Hyperbolic distance supplies a principled rule for choosing the height at which two tokens interact: nearby tokens interact at fine scales, while far-apart tokens interact through coarse representatives.

Formulas

$$d_{S}\bigl((x,y),(x',y')\bigr)=C_{N}\,\operatorname{arcosh}\!\left(1+\frac{d_{N}(x,x')^{2}+(y-y')^{2}}{2yy'}\right)$$

$$h(\gamma)=\max_{0\leq j\leq m}\mathcal{h}(z_j)$$

$$Q\subset Q_h,\qquad R\subset R_h,\qquad Q_h,R_h\in\mathscr{D}_h$$

$$\ell^{*}(i,j)=\arg\min_{\ell\in\{0,\ldots,L\}}\left|d_{S}\bigl((x_i,y_\ell),(x_j,y_\ell)\bigr)-\tau_\ell\right|$$

Mathematical statement

The paper defines the hyperbolic-type half-space metric on S=N imes\mathbb{R}_{+} by d_{S}((x,y),(x',y'))=C_{N}\operatorname{arcosh}(1+(d_{N}(x,x')^{2}+(y-y')^{2})/(2yy')), where x,x' are base-space positions, y,y'>0 are heights, d_N is the base metric, and C_N=(\log\eta)^{-1} is a normalization. It discretizes S with dyadic cells Q and a unique parent operation. For a path \gamma=(z_0,\ldots,z_m) in the spiderweb graph, its height is h(\gamma)=\max_{0\le j\le m}\mathcal{h}(z_j). Lemma 3.5 states that for endpoints z_Q,z_R and any path of height h, there are unique ancestors Q_h,R_h at level h satisfying Q\subset Q_h and R\subset R_h. The standard-geodesic construction uses ascent through parent edges, a uniformly bounded horizontal segment between same-level ancestors, and descent. The neural adaptation uses the same ancestor routing: every token is assigned to a leaf cell Q, pooled to ancestors Q_l, performs sparse horizontal attention between a bounded neighbor set at level l, and broadcasts back downward. The arcosh metric determines the preferred level l for a token pair, while the bounded horizontal degree controls compute.

Implementation notes

Integrate this as a replacement for one or more dense self-attention blocks in a Transformer encoder. For a sequence of n tokens, create a balanced dyadic hierarchy with levels l=0,...,L, where level 0 contains individual tokens and level L contains one or a few global cells. Let Q_l(i) be token i's ancestor at level l, let y_l=y_0\eta^l be its height, and let x_i=i/n for a sequence or the normalized 2D coordinate for an image. At each level, compute cell summaries with learned pooling: s_{Q_l}=\operatorname{Pool}({h_i:Q_l(i)=Q_l}), and construct keys and values from these summaries. For each cell Q at level l, define a fixed-size horizontal neighbor set N_l(Q), obtained from nearby cells in the base coordinate; use attention only between Q and N_l(Q). Finally broadcast the updated summary to descendants through learned gated residuals.

Pseudocode: (1) assign every token i to leaf cell Q_0(i); (2) for l=1,...,L, set Q_l(i)=parent(Q_{l-1}(i)) and compute s_{Q_l}; (3) for every level l and cell Q, compute q_Q,k_R,v_R for R in N_l(Q), and set a_{QR}=softmax_R(q_Q^Tk_R/\sqrt d+b_{QR}); (4) update s'_Q=\sum_{R\in N_l(Q)}a_{QR}v_R; (5) propagate s'_Q to all descendant tokens and combine with the local representation using h_i\leftarrow h_i+g_l(i)s'_{Q_l(i)}. The paper's mathematics supplies the parent hierarchy, height-based routing, and hyperbolic distance. Estimate the base distance d_N from token-coordinate distance; estimate or tune \eta, y_0, neighbor radius, and gates empirically. Initially use all levels, then prune levels whose attention mass is negligible.

The first experiment should use a 2K- or 4K-token language-modeling Transformer on WikiText-103 or a synthetic long-range-copy task. Compare dense attention, local-window attention with the same hidden size, and Spiderweb attention at matched parameter count. Measure validation perplexity at equal training FLOPs, wall-clock throughput, peak activation memory, and performance on dependencies longer than the local window. A successful result is near-dense perplexity with O(n\log n) rather than O(n^2) attention work, at least 1.5x throughput or substantially lower memory, and improved accuracy over a purely local baseline on long-range-copy examples. An ablation must compare learned routing against routing based directly on the hyperbolic distance formula.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

How to read this: The implementation beat the baseline, but the claimed mechanism did not reproduce at benchmark scale. The gain may come from another effect; treat the cause as unresolved.

Stage 1 — Mechanism check agent confidence 8/10

Built a readable NumPy Spiderweb hierarchy with dyadic pooling, bounded same-level neighborhoods, attention updates, and downward broadcast. The math checks confirmed all three targeted predictions: equal-height hyperbolic distance decreased monotonically across levels, the inverse-metric transition level matched the discrete argmin with 96% exact agreement and mean error 0.04 levels, and normalized work remained bounded while dense work grew by 29.1x over the tested range. The impulse test showed global communication (far-token influence 0.00638 versus zero for local attention), while estimated work was 3,835 versus 65,536 dense pair scores; however, the unoptimized NumPy implementation was slower wall-clock than dense attention, so the demonstrated win is structural work/memory and long-range reach rather than runtime.

Agent confidence: 8/10

Baseline: Dense: 65,536 pair scores, 8.97e-5 s/call, far impulse influence 0.003906. Local: 1,280 pair scores, 0.00854 s/call, far impulse influence 0.
Idea: Spiderweb: estimated 3,835 operations (1,531 horizontal + 2,304 broadcast), 0.1151 s/call in unoptimized NumPy, far impulse influence 0.00637899; O(n log n) normalized work ranged 2.37 to 1.64.

Limitations: No learned Transformer, language-model perplexity, WikiText/long-range-copy training, matched parameter/FLOP optimization, GPU kernel, peak memory measurement, or hyperbolic-distance routing ablation was tested. The runtime comparison is not favorable because the reference implementation uses slow Python/NumPy loops and is not an optimized sparse kernel.

How to run: python3 spiderweb_experiment.py && /home/maxwelhelp/main/bin/python3 run_benchmark.py && /home/maxwelhelp/main/bin/python3 impulse_check.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.02369) Sep 2, 2026

Beats the tuned baseline by 24.7% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.1653
Idea mean: 0.1244
Effect (Δ): -0.0409 (−24.7%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±3.9%
Mechanism: Not confirmed ✗
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in sequence forecast track was tested with sequence length 32, 2,000 training examples, 500 test examples, and 15 epochs. No 2K/4K language modeling, long-range-copy task, wall-clock throughput, peak activation memory, optimized sparse GPU kernel, or hyperbolic-distance routing ablation was evaluated. The mechanism signature used a single held-out test example and was negative despite the task-metric win.

How to run: python3 bench_stage2.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Worked ✓

Agent confidence: 8/10

Spiderweb Hierarchical Attention was implemented as a separately trained, parameter-matched replacement for the registered bench transformer_tiny model on the structurally appropriate sequence forecast track. Across 8 paired seeds it reduced test MSE from 0.1653229 to 0.1244110, with all 8 wins and permutation p=0.0081, satisfying the bench criterion. The trained-model long-range influence signature was not confirmed, and throughput/memory were not measured.

Baseline: Swept dense transformer_tiny; best lr=0.001; full 8-seed test MSE mean 0.16532287374138832, std 0.008160238971414253.
Comparison: delta_mean=-0.040911887772381306, idea_wins=8, p_value=0.0081, verdict=idea better (significant).
Idea: Swept spiderweb transformer over the same learning-rate union; best lr=0.001; 8-seed test MSE mean 0.12441098596900702, std 0.008560845180504158.

Benchmark result