SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformers

What the math gives to ML

The paper provides a lightweight rule for adapting the training-only slope of a spiking neuron's surrogate derivative from transformer attention uncertainty. Its transferable asset is the use of normalized attention entropy as a block-local difficulty signal, combined with a dead zone that prevents noisy updates and a bounded monotone response through tanh. The most direct implementation is to compute one uncertainty statistic per transformer block, map it to a surrogate-gradient temperature or slope, and leave the forward spikes and inference graph unchanged. This is a moderate but concrete idea for improving optimization of spiking transformers, especially when different blocks exhibit substantially different attention concentration.