# First-Spike Laplacian Attention

- ID: 2152
- Canonical URL: https://synthcore.org/idea/2152/first-spike-laplacian-attention
- API JSON: https://synthcore.org/api/idea/2152.json
- API Markdown: https://synthcore.org/api/idea/2152.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.11865](https://arxiv.org/abs/2608.11865)
- Category: architecture
- Solves: speedup, stability, accuracy
- ML areas: attention, transformer, inference-speedup
- Math tags: metric-geometry, dynamical-systems, linear-algebra
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace multiplicative query-key attention scores with an affinity based on the l1 distance between first-spike latency vectors. For each query token and key token, small latency differences produce large affinity and distant timings decay exponentially, yielding a locality-sensitive attention pattern naturally compatible with leaky spiking neurons.

## Mathematical statement

For query token i and key token j, let q_i,k_j in R^C be their first-spike latency vectors across C channels, measured from the beginning of a coding window. The latency distance is d_ij=||q_i-k_j||_1=sum_c |q_ic-k_jc|. The Laplacian affinity is a_ij=exp(-d_ij/sigma), where sigma>0 is a bandwidth or latency scale. Row-normalized attention is A_ij=a_ij/sum_l a_il, with N key tokens. The physical motivation follows the LIF dynamics tau_m du(t)/dt=-(u(t)-u_rest)+I(t), whose zero-input solution is u(t+Delta)-u_rest=(u(t)-u_rest) exp(-Delta/tau_m). Thus sigma can be initialized from the membrane time constant tau_m or learned as sigma=softplus(theta)+epsilon. The transferable properties are the nonnegative channel-separable metric and monotone exponential decay; no query-key channel multiplications are required.

## Key formulas

- $$\tau_m\frac{\mathrm{d}u(t)}{\mathrm{d}t}=-(u(t)-u_{\mathrm{rest}})+I(t)$$
- $$u(t+\Delta)-u_{\mathrm{rest}}=\bigl(u(t)-u_{\mathrm{rest}}\bigr)\exp\left(-\frac{\Delta}{\tau_m}\right)$$
- $$d_{ij}=\lVert q_i-k_j\rVert_1=\sum_{c=1}^{C}\left|q_{ic}-k_{jc}\right|,\qquad a_{ij}=\exp\left(-\frac{d_{ij}}{\sigma}\right)$$
- $$A_{ij}=\frac{a_{ij}}{\sum_{\ell=1}^{N}a_{i\ell}},\qquad y_i=\sum_{j=1}^{N}A_{ij}v_j$$

## Implementation notes

Integrate this at the attention score stage of a spiking transformer, after the Q and K projection layers and before value aggregation. Inputs are latency tensors q and k with shape [batch, heads, tokens, channels], where each element is the first spike time in a coding window; use a sentinel equal to the window length for channels that never spike. For every query-key pair, compute D[i,j]=sum_c abs(q[i,c]-k[j,c]) using broadcasted subtraction, absolute value, and reduction. Then calculate A=softmax(-D/sigma, dim=-1) and output Y=A@V. Use sigma=softplus(theta)+epsilon, with one learned theta per head, or initialize sigma to the median pairwise l1 distance divided by log(2). During the first experiment, use floating-point arithmetic and a differentiable latency surrogate. Compare against scaled dot-product attention in a 4-8 layer spiking ViT on CIFAR-10 or CIFAR-100, keeping Q/K/V widths, encoder, optimizer, and schedule identical. Measure top-1 accuracy, Q-K multiply count, wall-clock latency, and an energy proxy. The target is accuracy within one percentage point of dot-product attention while eliminating nearly all query-key channel multiplications and reducing attention-path energy.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built First-Spike Laplacian attention with differentiable per-head positive bandwidths, L1 latency distances, stable softmax normalization, and value aggregation in laplacian_attention.py. The mechanism checks matched theory: log affinity ratios had maximum error 8.9e-16, the two-key probability transition crossed 0.5 at zero distance gap with kernel-level error 3.4e-9, and entropy increased monotonically with sigma toward log(N); forward normalization and gradients were finite. In the toy retrieval comparison, Laplacian attention achieved better MSE and substantially better top-1 retrieval at low-to-moderate noise, but this is not evidence of CIFAR or wall-clock superiority.

### Mechanism check

- Verdict: Built First-Spike Laplacian attention with differentiable per-head positive bandwidths, L1 latency distances, stable softmax normalization, and value aggregation in laplacian_attention.py. The mechanism checks matched theory: log affinity ratios had maximum error 8.9e-16, the two-key probability transition crossed 0.5 at zero distance gap with kernel-level error 3.4e-9, and entropy increased monotonically with sigma toward log(N); forward normalization and gradients were finite. In the toy retrieval comparison, Laplacian attention achieved better MSE and substantially better top-1 retrieval at low-to-moderate noise, but this is not evidence of CIFAR or wall-clock superiority.
- Confidence: 9/10
- Limitations: Only a synthetic latency-code retrieval task was tested; no spiking ViT, CIFAR accuracy, learned bandwidth training study, CUDA benchmark, wall-clock latency, or hardware energy measurement was performed. The operation comparison counts arithmetic categories but does not establish actual runtime or energy savings.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.63405
- Paired wins: 3/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/863/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/863/bench_report.json)
- [experiment.py](https://synthcore.org/code/863/experiment.py)
- [laplacian_attention.py](https://synthcore.org/code/863/laplacian_attention.py)
- [report.md](https://synthcore.org/code/863/report.md)
- [report_bench_2026-09-01T125658.md](https://synthcore.org/code/863/report_bench_2026-09-01T125658.md)
- [verify_module.py](https://synthcore.org/code/863/verify_module.py)
- [Download all files as ZIP](https://synthcore.org/download/863)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.