Channel-aware attention-head pruning

(1) Integration point: use this as an offline head-selection algorithm before fine-tuning a Transformer, then physically remove the selected Q, K, V, and output-projection slices for those heads. Collect attention probabilities from 1,000 to 10,000 calibration sequences before pruning, using all query positions with at least two valid keys. (2) Pseudocode: initialize pairwise accumulators A_content and A_sink. For each batch, layer, head, and valid query row, extract s=attention[...,0], compute c=attention[...,1:]/(1-s), clamp c to eps and renormalize, compute z(c)=log(c)-mean(log(c)), and compute u(s)=logit(clamp(s,eps,1-eps)). For every head pair, accumulate squared z differences divided by m and squared u differences divided by 2. Form score_ab=alpha A_content[a,b]+(1-alpha) A_sink[a,b], but allow a merge only if both A_content[a,b] is below tau_content and A_sink[a,b] is below tau_sink. If a pair is admissible, retain the head with lower measured ablation importance, where importance is the validation-loss increase or output perturbation caused by zeroing that head. (3) The channel decomposition and distance formulas come from the paper; alpha, thresholds, epsilon, and the calibration distribution are estimated empirically. (4) First experiment: prune 25%, 50%, and 75% of heads in BERT-base or a 6-layer decoder on WikiText-2. Compare random pruning, raw-row cosine pruning, Jensen-Shannon pruning, and the two-channel method, followed by identical short recovery fine-tuning. Measure retained parameters, attention FLOPs, latency, and perplexity. Success is lower perplexity at the same retained head count and a larger gap over raw-row metrics when sink mass is high. Also verify that selected redundant pairs have low content distance rather than merely similar sink probabilities.