Entropy-Gated Surrogate Slopes
Implementation & benchmark of arXiv:2608.13702 — SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformers
Source paper: SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformers arXiv:2608.13702 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Adapt the slope of each spiking neuron's surrogate derivative using the normalized entropy of its block's attention distribution. High centered entropy uncertainty increases the slope, while low uncertainty decreases it, and a dead zone holds the default slope fixed for ordinary fluctuations. The adaptation exists only in backpropagation, so the forward spike function, parameter count, and inference cost remain unchanged.
Formulas
Mathematical statement
Proposition 1 defines the adaptive slope for a centered normalized entropy-dispersion statistic z_c as alpha(z_c)=alpha_0 when |z_c|<delta and alpha_0+beta tanh(z_c) otherwise, with alpha_0>0, beta>0, and delta>0. Here alpha_0 is the baseline surrogate-gradient slope, beta is the maximum signed adaptation magnitude, delta suppresses small uncertainty changes, and tanh bounds the adaptation. The proposition proves monotonicity outside the dead zone because d alpha/d z_c = beta(1-tanh^2(z_c))>0. To make z_c computable, let an attention block have heads h=1,...,H, query positions i=1,...,L, and attention probabilities p_{h,i,j} over keys j=1,...,L. Define normalized entropy u_{h,i}=-(1/log L) sum_{j=1}^L p_{h,i,j} log(p_{h,i,j}+epsilon), then block uncertainty u=(HL)^(-1) sum_{h,i}u_{h,i}. Maintain an exponential running mean m and standard deviation s, and use z_c=(u-m)/(s+epsilon). The surrogate derivative can be instantiated as the derivative of a logistic spike surrogate, g_alpha(v)=alpha sigmoid(alpha v)(1-sigmoid(alpha v)), where v is membrane voltage minus threshold; alpha is detached from the computation graph and selected by the proposition.
Implementation notes
1. Integration point: modify the custom autograd function for the spike operation in each spiking transformer block. The forward pass must remain S=(v>=theta).float(). During the backward pass, use the block's scalar alpha instead of a globally fixed surrogate slope. Compute attention entropy immediately after softmax(QK^T/sqrt(d)); do not backpropagate through entropy or alpha. 2. Pseudocode: for each training batch and block b, compute P=softmax(logits,dim=-1); u_b=mean(-sum(P*log(P+eps),dim=-1)/log(L)); update running statistics m_b=(1-rho)m_b+rho*u_b and q_b=(1-rho)q_b+rho*u_b^2; s_b=sqrt(max(q_b-m_b^2,0)+eps); z_b=(u_b-m_b)/(s_b+eps); if abs(z_b)<delta set alpha_b=alpha0, else alpha_b=alpha0+beta*tanh(z_b); clamp alpha_b to [alpha_min,alpha_max]; call the spike autograd function with alpha_b. In backward, return upstream_gradient multiplied by alpha_b*sigmoid(alpha_b*v)*(1-sigmoid(alpha_b*v)). 3. The paper supplies the bounded monotone mapping and dead-zone logic. The entropy definition, running moments, clipping, and logistic surrogate are implementation choices that make the statistic operational; compare them against the paper's exact surrogate if available. Use rho around 0.01-0.05, alpha_min=max(0.1,alpha0-beta), and alpha_max=alpha0+beta. 4. First experiment: train a small 4-block spiking ViT on CIFAR-10 for 4 or 8 simulation steps using the same optimizer, seed set, and fixed-slope surrogate baseline. Test alpha0 in {1,2,4}, beta in {0.25,0.5,1}, and delta in {0.25,0.5,1}. Measure validation accuracy, loss after equal optimizer steps, gradient norm variance per block, firing rate, and wall-clock time. The expected signal is faster early loss descent or 0.5-1% higher accuracy at equal compute, with no measurable inference slowdown and reduced gradient instability in blocks whose attention entropy changes sharply.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.