# Dyadic Hankel Boundary Attention

- ID: 1303
- Canonical URL: https://synthcore.org/idea/1303/dyadic-hankel-boundary-attention
- API JSON: https://synthcore.org/api/idea/1303.json
- API Markdown: https://synthcore.org/api/idea/1303.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.23016](https://arxiv.org/abs/2607.23016)
- Category: architecture
- Solves: speedup, memory, scalability
- ML areas: attention, transformer, inference-speedup, kv-cache
- Math tags: functional-analysis, spectral-theory, approximation-theory, harmonic-analysis, numerical-analysis
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace dense attention between tokens on opposite sides of a one-dimensional boundary or segment split with a dyadic low-rank approximation of a Cauchy/Hankel distance kernel. Each distance-scale block uses O(log(1/\varepsilon)) features, and the number of active scales grows only logarithmically with context length after discarding a narrow boundary layer. This is especially suitable for a relative-position attention branch or state-space-like long-range branch, rather than arbitrary content-dependent softmax attention.

## Mathematical statement

The paper studies S=P_A Q_B P_A, where P_E is multiplication by the indicator of E, Q_E=\mathcal F^{-1}P_E\mathcal F, and A=cA_0, B=B_0. The nontrivial interaction is the off-diagonal operator T=P_{A^c}Q_BP_A. In one dimension, boundary-distance coordinates split every one-sided, one-scale piece into two summands carrying the singular values of the fixed Hankel kernel K(s,r)=1/(2\pi(s+r)), with s,r>0. For the restriction s\in(0,\infty), r\in[h,2h], the Bernstein-ellipse argument gives s_{N+1}(K)\leq4^{-N}, where s_j(K) is the j-th singular value and h>0 is the dyadic scale. Ky Fan assembly gives the scale-uniform piece estimate s_n\leq2^{3-n}. The global plunge estimate is \Lambda_\varepsilon\leq C(A_0,B_0)\widetilde L[1+\ln_+(ca/\widetilde L)], with \widetilde L=\ln[1/(\varepsilon(1-\varepsilon))], a the largest component length of A_0, and \Lambda_\varepsilon the number of eigenvalues of S in (\varepsilon,1-\varepsilon). The adaptation treats each attention cross-block as a discretized K(s,r); the singular-value decay implies rank N\geq\lceil\log_4(1/\varepsilon)\rceil for operator-norm error at most \varepsilon per dyadic block.

## Key formulas

- $$S=S_{cA_0,B_0}=P_AQ_BP_A,\qquad A=cA_0,\quad B=B_0,$$
- $$K(s,r)=\frac{1}{2\pi(s+r)},\qquad s>0,\ r\in[h,2h],$$
- $$s_{N+1}(K)\leq 4^{-N},\qquad\text{and hence for a one-scale piece}\qquad s_n\leq 2^{3-n},$$
- $$\Lambda_\varepsilon\leq C(A_0,B_0)\,\widetilde L\left(1+\ln_+\frac{ca}{\widetilde L}\right),\qquad \widetilde L=\ln\frac{1}{\varepsilon(1-\varepsilon)}.$$

## Implementation notes

Integrate this as an additional long-range attention branch for a sequence of length L, not as a replacement for all content-dependent softmax attention. Choose one or more fixed split points b; for every token i<b and j\geq b, define positive distances s_i=b-i+\delta and r_j=j-b+\delta, with \delta\geq1 preventing a singularity. Partition the r-coordinates into dyadic blocks B_k=[h_k,2h_k). For every block, form the sampled matrix H_k[i,j]=1/(s_i+r_j), optionally multiply it by a learned scalar gate and value projections. Compute a thin rank-N factorization H_k approximately equal to U_k V_k^    op, using N=\lceil\log_4(1/\varepsilon)\rceil from the paper's s_{N+1}\leq4^{-N} estimate; in practice, obtain U_k,V_k once by randomized SVD or Chebyshev interpolation and cache them. During inference, compute the cross-block output as U_k(V_k^    op V_tokens), costing O((|I_k|+|J_k|)N d_v) rather than O(|I_k||J_k|d_v). Add the reverse direction using the transposed factorization. The paper supplies the geometric rank rule and logarithmic scale count; empirical SVD on the discretized block estimates the effects of finite token spacing, \delta, normalization, and truncation. Start with a 4-layer, 8-head causal Transformer on TinyStories or WikiText-103 at contexts 2K, 8K, and 32K. Compare dense attention, standard local-plus-global attention, and this branch at matched validation perplexity and FLOPs. Success means at least 2x lower cross-block attention cost or KV memory at 8K/32K with less than 1% perplexity degradation, plus measured singular-value decay close to the predicted geometric curve.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a reproducible dyadic Hankel MVP in dyadic_hankel.py with cached thin-SVD application and quantitative checks. Across h=4..64, every tested normalized singular value satisfied the predicted s_(N+1) <= 4^-N bound; observed ranks were 2,3,3 for eps=1e-2,1e-3,1e-4 versus conservative bounds 4,5,7 and remained independent of h. Rank-4 factorization produced about 1e-6 relative output error and measured speedups of 2.1x-11.5x as block size grew, confirming the mechanism but not a full language-model win.

### Mechanism check

- Verdict: Built a reproducible dyadic Hankel MVP in dyadic_hankel.py with cached thin-SVD application and quantitative checks. Across h=4..64, every tested normalized singular value satisfied the predicted s_(N+1) <= 4^-N bound; observed ranks were 2,3,3 for eps=1e-2,1e-3,1e-4 versus conservative bounds 4,5,7 and remained independent of h. Rank-4 factorization produced about 1e-6 relative output error and measured speedups of 2.1x-11.5x as block size grew, confirming the mechanism but not a full language-model win.
- Confidence: 8/10
- Limitations: Only a CPU toy discretization of one relative-position Hankel cross-block was tested. No learned gates, reverse-direction integration, multi-block attention, GPU kernel implementation, Transformer training, perplexity, KV-cache memory, or context lengths beyond the toy block sizes were evaluated; timing is sensitive to BLAS and allocation effects.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.53235
- Paired wins: 5/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/516/bench_report.json)
- [dyadic_hankel.py](https://synthcore.org/code/516/dyadic_hankel.py)
- [report.md](https://synthcore.org/code/516/report.md)
- [report_bench_2026-08-31T164330.md](https://synthcore.org/code/516/report_bench_2026-08-31T164330.md)
- [results.json](https://synthcore.org/code/516/results.json)
- [stage2_bench.py](https://synthcore.org/code/516/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/516)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.