△ Mechanism confirmed, baseline not beaten 2026

Dyadic Hankel Boundary Attention

Implementation & benchmark of arXiv:2607.23016 — An independent proof of the plunge-region conjecture for time-frequency localization operators in dimension one

Usefulness7/10

Difficulty5/10

Novelty6/10

Architecture Speedup Memory Scalability Attention Transformer Inference speedup Kv cache Functional analysis Spectral theory Approximation theory Harmonic analysis Numerical analysis

Source paper: An independent proof of the plunge-region conjecture for time-frequency localization operators in dimension one arXiv:2607.23016 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace dense attention between tokens on opposite sides of a one-dimensional boundary or segment split with a dyadic low-rank approximation of a Cauchy/Hankel distance kernel. Each distance-scale block uses O(log(1/\varepsilon)) features, and the number of active scales grows only logarithmically with context length after discarding a narrow boundary layer. This is especially suitable for a relative-position attention branch or state-space-like long-range branch, rather than arbitrary content-dependent softmax attention.

Formulas

$$S=S_{cA_0,B_0}=P_AQ_BP_A,\qquad A=cA_0,\quad B=B_0,$$

$$K(s,r)=\frac{1}{2\pi(s+r)},\qquad s>0,\ r\in[h,2h],$$

$$s_{N+1}(K)\leq 4^{-N},\qquad\text{and hence for a one-scale piece}\qquad s_n\leq 2^{3-n},$$

$$\Lambda_\varepsilon\leq C(A_0,B_0)\,\widetilde L\left(1+\ln_+\frac{ca}{\widetilde L}\right),\qquad \widetilde L=\ln\frac{1}{\varepsilon(1-\varepsilon)}.$$

Mathematical statement

The paper studies S=P_A Q_B P_A, where P_E is multiplication by the indicator of E, Q_E=\mathcal F^{-1}P_E\mathcal F, and A=cA_0, B=B_0. The nontrivial interaction is the off-diagonal operator T=P_{A^c}Q_BP_A. In one dimension, boundary-distance coordinates split every one-sided, one-scale piece into two summands carrying the singular values of the fixed Hankel kernel K(s,r)=1/(2\pi(s+r)), with s,r>0. For the restriction s\in(0,\infty), r\in[h,2h], the Bernstein-ellipse argument gives s_{N+1}(K)\leq4^{-N}, where s_j(K) is the j-th singular value and h>0 is the dyadic scale. Ky Fan assembly gives the scale-uniform piece estimate s_n\leq2^{3-n}. The global plunge estimate is \Lambda_\varepsilon\leq C(A_0,B_0)\widetilde L[1+\ln_+(ca/\widetilde L)], with \widetilde L=\ln[1/(\varepsilon(1-\varepsilon))], a the largest component length of A_0, and \Lambda_\varepsilon the number of eigenvalues of S in (\varepsilon,1-\varepsilon). The adaptation treats each attention cross-block as a discretized K(s,r); the singular-value decay implies rank N\geq\lceil\log_4(1/\varepsilon)\rceil for operator-norm error at most \varepsilon per dyadic block.

Implementation notes

Integrate this as an additional long-range attention branch for a sequence of length L, not as a replacement for all content-dependent softmax attention. Choose one or more fixed split points b; for every token i<b and j\geq b, define positive distances s_i=b-i+\delta and r_j=j-b+\delta, with \delta\geq1 preventing a singularity. Partition the r-coordinates into dyadic blocks B_k=[h_k,2h_k). For every block, form the sampled matrix H_k[i,j]=1/(s_i+r_j), optionally multiply it by a learned scalar gate and value projections. Compute a thin rank-N factorization H_k approximately equal to U_k V_k^ op, using N=\lceil\log_4(1/\varepsilon)\rceil from the paper's s_{N+1}\leq4^{-N} estimate; in practice, obtain U_k,V_k once by randomized SVD or Chebyshev interpolation and cache them. During inference, compute the cross-block output as U_k(V_k^ op V_tokens), costing O((|I_k|+|J_k|)N d_v) rather than O(|I_k||J_k|d_v). Add the reverse direction using the transposed factorization. The paper supplies the geometric rank rule and logarithmic scale count; empirical SVD on the discretized block estimates the effects of finite token spacing, \delta, normalization, and truncation. Start with a 4-layer, 8-head causal Transformer on TinyStories or WikiText-103 at contexts 2K, 8K, and 32K. Compare dense attention, standard local-plus-global attention, and this branch at matched validation perplexity and FLOPs. Success means at least 2x lower cross-block attention cost or KV memory at 8K/32K with less than 1% perplexity degradation, plus measured singular-value decay close to the predicted geometric curve.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a reproducible dyadic Hankel MVP in dyadic_hankel.py with cached thin-SVD application and quantitative checks. Across h=4..64, every tested normalized singular value satisfied the predicted s_(N+1) <= 4^-N bound; observed ranks were 2,3,3 for eps=1e-2,1e-3,1e-4 versus conservative bounds 4,5,7 and remained independent of h. Rank-4 factorization produced about 1e-6 relative output error and measured speedups of 2.1x-11.5x as block size grew, confirming the mechanism but not a full language-model win.

Agent confidence: 8/10

Baseline: Dense H@V: 0.077-1.568 ms for h=64-512; relative error 0 by definition; arithmetic cost ratio versus rank-4 factorization is 12.8x-102.4x.
Idea: Rank-4 cached factorization: 0.036-0.137 ms, relative output error 1.0e-6-1.9e-6, measured speedup 2.12x-11.46x. Singular-bound checks passed 25/25; observed ranks for eps=(1e-2,1e-3,1e-4) were (2,3,3) at every h.

Limitations: Only a CPU toy discretization of one relative-position Hankel cross-block was tested. No learned gates, reverse-direction integration, multi-block attention, GPU kernel implementation, Transformer training, perplexity, KV-cache memory, or context lengths beyond the toy block sizes were evaluated; timing is sensitive to BLAS and allocation effects.

How to run: python3 dyadic_hankel.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.23016) Sep 2, 2026

Ahead of the baseline (−0.8%) but not statistically significant (p=0.5323, wins 5 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.3629
Idea mean: 0.36
Effect (Δ): -0.003 (−0.8%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.5323 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±2.9%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the registered sequence forecast track was tested at window length 32, 400 training samples, 200 test samples, and 20 epochs. Wall-clock speed, FLOPs, KV-cache memory, longer contexts, multiple dyadic scales, learned gates, causal masking, and language-model perplexity were not measured.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

The registered built-in sequence track was rerun end-to-end with matched transformer systems, an 8-seed paired protocol, tuned baseline learning-rate sweep, and a three-setting idea rank sweep. The dyadic system slightly improved mean test MSE, but the paired permutation test was non-significant, so there is no demonstrated benchmark win. The trained-model mechanism signature was confirmed quantitatively: observed rank 4 <= predicted bound 5 and observed relative cross-output error 3.26e-05 <= 1e-3.

Baseline: sequence/transformer_tiny, tuned lr=0.001; 8-seed mean test MSE=0.3629476018, std=0.0543452814.
Idea: rank=4 at shared lr=0.001; 8-seed mean test MSE=0.3599541374, std=0.0551822606; paired delta=-0.0029934645, 5/8 wins, permutation p=0.53235.

Benchmark result