✓✓ Beats tuned baseline 2026

Critical-Tail Multiscale Mixer

Implementation & benchmark of arXiv:2607.04971 — Long-range interactions and Anderson localisation for one-dimensional high-contrast resonator chain

Usefulness6/10

Difficulty5/10

Novelty6/10

Architecture Accuracy Scalability Memory Ssm Attention Inference speedup Harmonic analysis Spectral theory Numerical analysis

Source paper: Long-range interactions and Anderson localisation for one-dimensional high-contrast resonator chain arXiv:2607.04971 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add a fixed or weakly parameterized residual mixer whose interaction between sequence positions at distance $r$ is proportional to $1/(r\log^2 r)$. Instead of truncating the kernel at a short radius, represent its heavy tail with dyadic distance bands and compute each band using prefix sums or block pooling, giving every token access to arbitrarily distant context at roughly $O(L\log L)$ cost.

Formulas

$$\mathcal{C}(n,m)\sim\frac{1}{|n-m|\log^{2}|n-m|},\qquad |n-m|\to\infty.$$

$$K(r)=\frac{1}{(r+r_0)\,[\log(r+r_0)]^2},\qquad r\geq1,\quad r_0>1.$$

$$\sum_{r>R}K(r)\approx\int_R^\infty\frac{dr}{r\log^2r}=\frac{1}{\log R}.$$

$$\int_{2^k}^{2^{k+1}}\frac{dr}{r\log^2r}=\frac{1}{k\log2}-\frac{1}{(k+1)\log2}=\frac{1}{k(k+1)\log2}.$$

Mathematical statement

The paper's central asymptotic result is $\mathcal{C}(n,m)\sim 1/(|n-m|\log^2|n-m|)$ as $|n-m|\to\infty$, where $n,m$ are resonator-chain indices and $\mathcal{C}(n,m)$ is an off-diagonal operator entry. The kernel is $\ell^1$-summable, so its total interaction mass is finite, but its tail is substantially longer than quadratic decay. For the neural adaptation, define $K(r)=1/((r+r_0)[\log(r+r_0)]^2)$ for integer distance $r\geq1$, with stabilizing constant $r_0>1$. Its omitted tail satisfies the integral estimate $\sum_{r>R}K(r)\approx\int_R^\infty dr/(r\log^2r)=1/\log R$, so short-range truncation converges unusually slowly. For a dyadic band $B_k=\{r:2^k\le r<2^{k+1}\}$, the band mass is approximately $\int_{2^k}^{2^{k+1}}dr/(r\log^2r)=1/(k\log2)-1/((k+1)\log2)=1/(k(k+1)\log2)$. These explicit masses determine multiscale mixing coefficients.

Implementation notes

Integrate the mixer after the token embedding or between Transformer/SSM blocks, operating on $X\in\mathbb{R}^{B\times L\times d}$. First project values with a small learned matrix $V=XW_v$. For each scale $k=0,\ldots,K-1$, where $2^K\le L$, compute left and right range sums over distances $[2^k,2^{k+1})$ using cumulative sums along the sequence; divide each range sum by its number of valid elements to obtain approximate band averages $P_k^\leftarrow$ and $P_k^\rightarrow$. Multiply the averages by the fixed coefficient $a_k=1/(k'(k'+1)\log2)$, with $k'=k+1$, and sum them. Use the residual update $Y=X+\alpha\,\mathrm{LN}(\sum_k a_k(P_k^\leftarrow+P_k^\rightarrow)W_o)$, where $\alpha$ and $W_o$ are learned; optionally add learned per-channel gates constrained positive with softplus. Pseudocode is: V=X@Wv; Y=0; prefix=cumsum(V,dim=sequence); for k in range(K): lo=2k; hi=2(k+1); left=(prefix[i-lo]-prefix[i-hi])/max(hi-lo,1); right=(prefix[i+hi]-prefix[i+lo])/max(hi-lo,1); Y += a[k]*(left+right); output=X+alpha*LN(Y@Wo). Boundary ranges must be clipped and renormalized. The mathematical quantities are the decay law, dyadic masses, and $1/\log R$ tail estimate; the learned quantities are $W_v,W_o,\alpha$, and optional channel gates. As a first experiment, train a 6-layer, 256-dimensional model on synthetic copy, reverse-retrieval, and associative-recall tasks plus Long Range Arena. Compare against depthwise convolution with matched parameters, a dilated convolution, and a linear-attention baseline. Measure validation accuracy versus context length, wall-clock time, peak memory, and gradient stability. The desired signal is better extrapolation beyond the training length with less memory than quadratic attention and better long-range retrieval than a fixed-radius convolution at equal FLOPs.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built a dyadic critical-tail mixer and numerical verification scripts. The discrete tail ratio to 1/log(R+2) approached 1.000 from 0.979 at R=8 to 0.99993 at R=1024, confirming the claimed slow tail decay; dyadic kernel approximation had lower L1 error than a radius-8 local kernel at every tested length. In the learned denoising experiment, validation MSE was 1.0341 versus 1.0389 at L=64 and 1.0162 versus 1.0278 at L=256 for dyadic versus local, with about 18% higher runtime at L=256, indicating a small long-context signal rather than a definitive accuracy win.

Agent confidence: 7/10

Baseline: Radius-8 local mixer: val MSE 1.03890 (L=64), 1.02779 (L=256); kernel L1 error 0.46367, 0.48266, 0.49304, 0.49955 at L=64,256,1024,4096; runtime 12.06s at L=256.
Idea: Dyadic tail mixer: val MSE 1.03412 (L=64), 1.01624 (L=256); kernel L1 error 0.25475, 0.25870, 0.25955, 0.25950 at L=64,256,1024,4096; runtime 14.27s at L=256. Tail ratio discrete/(1/log(R+2)) was 0.97896 at R=8 and 0.99993 at R=1024.

Limitations: The experiment is a one-dimensional scalar denoising task, not Transformer-scale copy, retrieval, or Long Range Arena. It uses a learned scalar residual rather than the full projected LN/Wv/Wo implementation, has only one seed and short training, and does not benchmark peak memory or compare against attention/linear attention. The dyadic bands use uniform averages, so the approximation is intentionally coarse.

How to run: python3 run_experiment.py && /home/maxwelhelp/main/bin/python3 verify_and_compare.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.04971) Sep 4, 2026

Beats the tuned baseline by 41.4% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.3913
Idea mean: 0
Effect (Δ): -0.1621 (−41.4%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±5.5%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in sequence forecast track was tested; CIFAR-10, tabular, and dynamics tracks were not tested. The comparison used the fixed tiny 32-token benchmark and 18 training epochs, not long-context extrapolation, peak-memory scaling, wall-clock scaling, copy/retrieval tasks, Long Range Arena, or a larger Transformer.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Worked ✓

Agent confidence: 8/10

Built the Critical-Tail Multiscale Mixer as a replacement for self-attention in transformer_tiny and evaluated it on the mandated sequence forecast track. Across 8 paired seeds, test MSE improved from 0.391339 for the tuned standard Transformer baseline to 0.229231 for the mixer; the paired permutation test gave delta_mean=-0.162108 and p=0.0081, so the effect is significant. A trained-model finite-perturbation signature also confirmed stronger distant influence for the mixer.

Baseline: Standard transformer_tiny self-attention; tuned lr=0.001, epochs=18; mean test MSE 0.391339, std 0.034275.
Idea: Critical-tail dyadic mixer; tuned lr=0.003, epochs=18; mean test MSE 0.229231, std 0.030839; paired delta=-0.162108; 8/8 seed wins; permutation p=0.0081.
Math sanity: Tail ratio to 1/log(R+2): R=8 0.97896, R=32 0.99586, R=128 0.99921, R=512 0.99984.
Mechanism signature: Baseline trained-model far/near influence ratio=4.4649; idea=12.6454; confirmed=true.

Benchmark result