Trainable two-sided activation gate

1. Integration point: start with the output activations of every large linear projection in an SSM block, including input, state, and output projections, or with transformer MLP and QKV output projections. Keep matrix weights unchanged initially; apply the gate immediately after the quantized matrix multiplication and before the next normalization or nonlinear operation. Store one trainable scalar $\alpha_j$ for each projection $j$; a second experiment can use one $\alpha_{j,k}$ per output channel. 2. Forward and backward pseudocode: compute $z=\operatorname{QuantLinear}_j(h)$; set $\Delta=\operatorname{softplus}(\alpha_j)+10^{-6}$; compute $y=\operatorname{sign}(z)\max(|z|-\Delta,0)$; during backward, replace the derivative with $g_z=\sigma(C(|z|-\Delta))$ and use $g_\Delta=-\operatorname{sign}(z)g_z$ for threshold learning. Implement this as a custom autograd function or straight-through estimator. Optionally add $\lambda\rho$ to the task loss, where $\rho$ is the fraction of values gated to zero, but first train without this term to measure natural sparsity. 3. Mathematics versus empirical estimates: the gate and surrogate derivatives come directly from the formulas above; $C$, threshold initialization, whether thresholds are per-projection or per-channel, and the sparsity penalty $\lambda$ are hyperparameters. Estimate $\rho$ on every validation batch and report it separately for every projection. 4. Cheap experiment: train a 100M-parameter S4/Mamba-style language model or small decoder-only transformer on WikiText-103 or C4 for the same number of tokens, comparing dense, fixed-threshold, and trainable-threshold gates. Initialize $\Delta$ at the activation 20th percentile so the initial zero rate is about 20%, then sweep target rates of 30%, 50%, and 75%. Measure validation perplexity, training loss at equal FLOPs, wall-clock time with a sparse kernel, fraction of zero activations, and energy if event-driven hardware is available. Success means at least 2x fewer nonzero activation events with less than 1% relative perplexity degradation, or faster loss descent at equal effective arithmetic operations.