Unverified 2026

Critical Heavy-Tail Propagation Layer

Usefulness6/10

Difficulty5/10

Novelty7/10

Architecture Scalability Accuracy Speedup Ssm Graph nn Attention Inference speedup Stochastic processes Dynamical systems Numerical analysis

Source paper: Propagation rates in integro-differential equations of ignition type arXiv:2608.28450 ⓘ · analyzed Sep 2, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Insert a sparse heavy-tailed nonlocal mixing operator into a residual sequence, graph, or spatial network so that information can traverse distant positions without stacking many local layers. Use the critical tail exponent s = 1/2, whose truncated first moment grows logarithmically and predicts an effective propagation distance proportional to depth times log depth rather than merely depth.

Formulas

$$M_1(R):=\int_{1}^{R} yJ(y)\,\mathrm{d}y$$

$$M_1(R)\asymp R^{1-2s}\quad\text{if }0<s<\frac{1}{2},\qquad M_1(R)\asymp\ln R\quad\text{if }s=\frac{1}{2}$$

$$X(t)\asymp t^{\frac{1}{2s}}\quad\text{if }0<s<\frac{1}{2},\qquad X(t)\asymp t\ln t\quad\text{if }s=\frac{1}{2}$$

$$h_i^{\ell+1}=h_i^{\ell}+\eta\sum_{k\in\mathcal K\setminus\{0\}}J_k\bigl(h_{i+k}^{\ell}-h_i^{\ell}\bigr),\qquad J_k=\frac{(1+|k|)^{-1-2s}}{\sum_{r\in\mathcal K\setminus\{0\}}(1+|r|)^{-1-2s}}$$

Mathematical statement

The paper defines the truncated first moment of a symmetric jump kernel J by M_1(R) = integral from 1 to R of y J(y) dy. For algebraic tails J(y) proportional to |y|^(-1-2s), it gives M_1(R) proportional to R^(1-2s) for 0 < s < 1/2, M_1(R) proportional to log R for s = 1/2, and bounded M_1(R) for s > 1/2. The associated propagation scales are linear for s > 1/2, X(t) proportional to t^(1/(2s)) for 0 < s < 1/2, and X(t) proportional to t log t at s = 1/2. Here R is a distance, t is the number of propagation steps, X(t) is the radius reached by a profile, and J is a nonnegative jump-weight density. The adaptation uses the discrete nonlocal generator D_J h_i = sum over k of J_k(h_(i+k) - h_i), with J_k proportional to (1 + |k|)^(-1-2s), and the residual update h^(ell+1) = h^ell + eta D_J h^ell. The mechanism is the divergent or logarithmic truncated first moment, which produces superlinear receptive-field growth while retaining a translation-invariant operator.

Implementation notes

1. Integration point: replace one or more local depthwise convolutions, graph message-passing steps, or fixed-window token-mixing layers with a nonlocal residual mixing block applied to hidden states h with shape batch by sequence length by channel dimension. For each token i, select symmetric offsets k from a fixed set K and apply the zero-sum operator D_J h_i = sum over nonzero k of J_k(h_(i+k) - h_i). Keep channel projections and nonlinearities unchanged. 2. Pseudocode: choose s = 0.5 and offsets k = plus or minus 1 through K. Set w_k = (1 + abs(k))^(-1 - 2s), normalize J_k = w_k divided by the sum of all w_r, compute mix = sum over k of J_k times (shift(h,k) - h), then update h = h + eta times mix. Mask offsets outside the sequence and renormalize surviving weights. 3. Compute J_k and the finite discrete moment M_1(R) = sum over 1 <= abs(k) <= R of abs(k) J_k analytically or once at initialization. For large sequences, sample offsets from the power law and use an unbiased estimator. 4. First experiment: compare a six-layer local depthwise-convolution SSM, fixed-window attention, and this heavy-tailed layer on copy, associative recall, and Long Range Arena tasks at equal parameters, training steps, and measured FLOPs. Test lengths 1k, 4k, and 16k. Measure validation accuracy, throughput, and gradient norms. 5. Pre-register mechanism tests: inject a one-hot impulse at layer zero and measure the distance containing 90 percent of absolute activation mass after t layers. The critical model should fit X_90(t) = a t log(1+t) + b with R-squared at least 0.95, while the local baseline should fit a linear function better. At equal depth and length 16k, require at least 1.5 times larger X_90 for the critical model. Repeat with s = 0.75 and s = 0.25; measured propagation should approach linear and quadratic scaling, respectively, with fitted exponent error below 20 percent. The falsification ablation replaces only the power-law weights with uniform weights over the same offsets. If long-range accuracy does not improve despite the predicted propagation signature, or if all exponents remain linear, reject the proposed mechanism.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.