△ Mechanism confirmed, baseline not beaten 2026

First-Spike Laplacian Attention

Implementation & benchmark of arXiv:2608.11865 — Lapis: Laplacian Spiking Attention via First-Spike Timing and Membrane Leakage

Usefulness7/10

Difficulty5/10

Novelty7/10

Architecture Speedup Stability Accuracy Attention Transformer Inference speedup Metric geometry Dynamical systems Linear algebra

Source paper: Lapis: Laplacian Spiking Attention via First-Spike Timing and Membrane Leakage arXiv:2608.11865 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace multiplicative query-key attention scores with an affinity based on the l1 distance between first-spike latency vectors. For each query token and key token, small latency differences produce large affinity and distant timings decay exponentially, yielding a locality-sensitive attention pattern naturally compatible with leaky spiking neurons.

Formulas

$$\tau_m\frac{\mathrm{d}u(t)}{\mathrm{d}t}=-(u(t)-u_{\mathrm{rest}})+I(t)$$

$$u(t+\Delta)-u_{\mathrm{rest}}=\bigl(u(t)-u_{\mathrm{rest}}\bigr)\exp\left(-\frac{\Delta}{\tau_m}\right)$$

$$d_{ij}=\lVert q_i-k_j\rVert_1=\sum_{c=1}^{C}\left|q_{ic}-k_{jc}\right|,\qquad a_{ij}=\exp\left(-\frac{d_{ij}}{\sigma}\right)$$

$$A_{ij}=\frac{a_{ij}}{\sum_{\ell=1}^{N}a_{i\ell}},\qquad y_i=\sum_{j=1}^{N}A_{ij}v_j$$

Mathematical statement

For query token i and key token j, let q_i,k_j in R^C be their first-spike latency vectors across C channels, measured from the beginning of a coding window. The latency distance is d_ij=||q_i-k_j||_1=sum_c |q_ic-k_jc|. The Laplacian affinity is a_ij=exp(-d_ij/sigma), where sigma>0 is a bandwidth or latency scale. Row-normalized attention is A_ij=a_ij/sum_l a_il, with N key tokens. The physical motivation follows the LIF dynamics tau_m du(t)/dt=-(u(t)-u_rest)+I(t), whose zero-input solution is u(t+Delta)-u_rest=(u(t)-u_rest) exp(-Delta/tau_m). Thus sigma can be initialized from the membrane time constant tau_m or learned as sigma=softplus(theta)+epsilon. The transferable properties are the nonnegative channel-separable metric and monotone exponential decay; no query-key channel multiplications are required.

Implementation notes

Integrate this at the attention score stage of a spiking transformer, after the Q and K projection layers and before value aggregation. Inputs are latency tensors q and k with shape [batch, heads, tokens, channels], where each element is the first spike time in a coding window; use a sentinel equal to the window length for channels that never spike. For every query-key pair, compute D[i,j]=sum_c abs(q[i,c]-k[j,c]) using broadcasted subtraction, absolute value, and reduction. Then calculate A=softmax(-D/sigma, dim=-1) and output Y=A@V. Use sigma=softplus(theta)+epsilon, with one learned theta per head, or initialize sigma to the median pairwise l1 distance divided by log(2). During the first experiment, use floating-point arithmetic and a differentiable latency surrogate. Compare against scaled dot-product attention in a 4-8 layer spiking ViT on CIFAR-10 or CIFAR-100, keeping Q/K/V widths, encoder, optimizer, and schedule identical. Measure top-1 accuracy, Q-K multiply count, wall-clock latency, and an energy proxy. The target is accuracy within one percentage point of dot-product attention while eliminating nearly all query-key channel multiplications and reducing attention-path energy.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built First-Spike Laplacian attention with differentiable per-head positive bandwidths, L1 latency distances, stable softmax normalization, and value aggregation in laplacian_attention.py. The mechanism checks matched theory: log affinity ratios had maximum error 8.9e-16, the two-key probability transition crossed 0.5 at zero distance gap with kernel-level error 3.4e-9, and entropy increased monotonically with sigma toward log(N); forward normalization and gradients were finite. In the toy retrieval comparison, Laplacian attention achieved better MSE and substantially better top-1 retrieval at low-to-moderate noise, but this is not evidence of CIFAR or wall-clock superiority.

Agent confidence: 9/10

Baseline: Scaled dot-product attention: retrieval top-1/MSE at noise 0.03, 0.10, 0.20, 0.35 = 0.0547/0.007728, 0.0156/0.007728, 0.0313/0.007732, 0.0859/0.007734.
Idea: First-Spike Laplacian attention: retrieval top-1/MSE at noise 0.03, 0.10, 0.20, 0.35 = 1.0000/0.007647, 0.9609/0.007672, 0.5234/0.007697, 0.1406/0.007718. For B=1,N=M=128,C=8, counted Q-K channel multiplications were 131072 for dot-product versus 0 for Laplacian, replaced by 131072 subtract/absolute operations.

Limitations: Only a synthetic latency-code retrieval task was tested; no spiking ViT, CIFAR accuracy, learned bandwidth training study, CUDA benchmark, wall-clock latency, or hardware energy measurement was performed. The operation comparison counts arithmetic categories but does not establish actual runtime or energy savings.

How to run: python3 experiment.py && /home/maxwelhelp/main/bin/python3 verify_module.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.11865) Sep 2, 2026

No significant difference vs. the tuned baseline (+1.7%, p=0.6341, wins 3 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.3238
Idea mean: 0.3293
Effect (Δ): 0.0055 (+1.7%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.6341 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±9.0%
Mechanism: Confirmed ✓
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in sequence forecasting track was tested; no CIFAR spiking ViT, actual first-spike event simulation, wall-clock kernel benchmark, GPU energy measurement, or larger/deeper transformer was evaluated. The latency representation used a differentiable sigmoid surrogate rather than discrete first-spike times.

How to run: cd /home/maxwelhelp/all/math2nn/experiments/exp2152_first-spike-laplacian-attention && /home/maxwelhelp/main/bin/python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented First-Spike Laplacian attention as an end-to-end replacement for scaled dot-product attention in the structurally matched sequence forecasting track, using identical transformer projections, feed-forward blocks, optimizer, epochs, and batch size. Across 8 paired seeds, the idea had higher MSE (0.3292590510 vs 0.3237519413; paired delta +0.0055071097) and was not significant (permutation p=0.63405), so there is no observed benchmark win. The trained-model mechanism signature was confirmed: attention rows were normalized and positive, with row-sum error 2.38e-7 and learned sigma values near initialization.

Baseline: Scaled dot-product transformer_tiny on sequence forecasting; best lr=0.003; 8-seed test MSE mean 0.3237519413, std 0.0466246347.
Baseline sweep: lr=0.0015: 0.3423246816; lr=0.003: 0.3276267201; lr=0.006: 0.5004829466 on 4 sweep seeds.
Idea: First-Spike Laplacian transformer_tiny using sigmoid latency surrogates, L1 distances, and learned positive bandwidth; best sigma=0.7 at lr=0.003; 8-seed test MSE mean 0.3292590510, std 0.0575468007.
Idea sweep: sigma=0.35: 0.3341145515; sigma=0.7: 0.3163300715; sigma=1.4: 0.3244466111 on 4 sweep seeds, all at lr=0.003.
Operation accounting: Per query-key channel, dot-product attention uses 32 multiplications; Laplacian attention uses 0 query-key channel multiplications and 32 subtract/absolute operations, excluding projections and reductions.
Paired comparison: delta_mean=+0.0055071097, idea_wins=3/8, permutation p=0.63405, verdict=no significant win.

Benchmark result