Unverified 2026

Power-of-Two Latency Attention

Implementation & benchmark of arXiv:2608.11865 — Lapis: Laplacian Spiking Attention via First-Spike Timing and Membrane Leakage

Usefulness6/10

Difficulty4/10

Novelty7/10

Quantization Speedup Memory Scalability Attention Quantization Inference speedup Transformer Numerical analysis Dynamical systems Linear algebra Metric geometry

Source paper: Lapis: Laplacian Spiking Attention via First-Spike Timing and Membrane Leakage arXiv:2608.11865 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Make Laplacian latency attention hardware-friendly by quantizing the exponential decay and row normalization to powers of two. Exponentials and divisions can then be implemented with lookup tables and integer shifts, enabling a low-bit attention kernel for neuromorphic and edge accelerators.

Formulas

$$a_{ij}=\exp\left(-\frac{d_{ij}}{\sigma}\right),\qquad d_{ij}=\sum_{c=1}^{C}|q_{ic}-k_{jc}|$$

$$r=\exp\left(-\frac{1}{\sigma}\right),\qquad \hat r=2^{-b},\qquad b=\operatorname{round}\left(\frac{1}{\sigma\ln 2}\right)$$

$$\hat a_{ij}=\hat r^{d_{ij}}=2^{-b d_{ij}},\qquad s_i=\sum_{j=1}^{N}\hat a_{ij},\qquad \hat A_{ij}=\hat a_{ij}\,2^{-\operatorname{round}(\log_2 s_i)}$$

$$y_i=\sum_{j=1}^{N}\hat A_{ij}v_j$$

Mathematical statement

Let d_ij=sum_c |q_ic-k_jc| be the nonnegative integer or fixed-point latency distance and let a_ij=exp(-d_ij/sigma). Define the decay per latency unit as r=exp(-1/sigma), with 0<r<1. Quantize it to a power of two, r_hat=2^{-b}, where b=round(1/(sigma ln 2)). Then a_hat_ij=r_hat^{d_ij}=2^{-b d_ij}, which is a fixed-point right shift by b d_ij bits. For each query row, let s_i=sum_j a_hat_ij. Approximate its reciprocal by a power of two, s_i^{-1} approximately 2^{-round(log_2 s_i)}, and use A_hat_ij=a_hat_ij 2^{-round(log_2 s_i)}. The approximation error is controlled by the rounding of b and the row-scale exponent. This construction transfers the paper's exponential membrane-decay interpretation into an integer-only score and normalization path.

Implementation notes

Modify the inference-time score and normalization kernel, not the Q/K projections or value path. Quantize first-spike latencies q and k to unsigned 6-bit or 8-bit integers representing coding-window ticks. Compute D with integer subtraction, absolute value, and channel accumulation: D=abs(Q[...,None,:]-K[...,None,:,:]).sum(-1). Select one decay shift b per head using b=round(1/(sigma*log(2))); compute raw affinity with a lookup table indexed by clipped b*D, or directly represent 2^{-bD} as a fixed-point integer right shift. For each query row, accumulate s=sum_j raw_affinity[j], estimate e=round(log2(s)) using leading-zero detection, and normalize using the corresponding left or right shift while retaining F fractional bits. During training, use exact floating-point Laplacian attention and fake-quantization or a straight-through estimator for latency, b, affinities, and values. Measure exact-versus-quantized rowwise KL divergence, output cosine error, accuracy, SRAM use, integer operation count, and accelerator energy. Start with a 4-layer spiking ViT on CIFAR-10 and compare float Laplacian, power-of-two Laplacian, and dot-product baselines at 6-bit and 8-bit precision. Seek less than 0.5 percentage-point accuracy loss and a substantial reduction in multiplications, divisions, and attention energy.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.