# Trainable two-sided activation gate

- ID: 2889
- Canonical URL: https://synthcore.org/idea/2889/trainable-two-sided-activation-gate
- API JSON: https://synthcore.org/api/idea/2889.json
- API Markdown: https://synthcore.org/api/idea/2889.md
- Verification status: unverified
- Source: [arXiv:2608.30439](https://arxiv.org/abs/2608.30439)
- Category: architecture
- Solves: speedup, memory, accuracy
- ML areas: ssm, transformer, inference-speedup, quantization
- Math tags: optimization, approximation-theory, dynamical-systems
- Ratings: usefulness 6/10; difficulty 4/10; novelty 5/10

## Idea description

Replace the dense output of selected linear projections with a two-sided magnitude threshold that emits zero for small values but preserves signed large values. Learn one positive threshold per projection, or optionally one threshold per output channel, so the network discovers where sparse events can be removed while retaining outlier information.

## Mathematical statement

The paper's sparse pre-activation is $x_{\mathrm{sparse}}=\operatorname{sign}(x)\operatorname{ReLU}(|x|-\Delta)$, where $x\in\mathbb{R}$ is a scalar pre-activation and $\Delta>0$ is a trainable threshold associated with a linear projection. Equivalently, $x_{\mathrm{sparse}}=0$ when $|x|\leq\Delta$, $x-\Delta$ when $x>\Delta$, and $x+\Delta$ when $x< -\Delta$; therefore both positive and negative outliers survive. The paper avoids completely zero gradients at the threshold using the surrogate derivative $\left(\frac{dx_{\mathrm{sparse}}}{dx}\right)_{\mathrm{smooth}}=\sigma(C(|x|-\Delta))$, where $\sigma(u)=1/(1+e^{-u})$ is the logistic sigmoid and $C>0$ controls transition sharpness. For a trainable threshold, use the corresponding surrogate $\partial x_{\mathrm{sparse}}/\partial\Delta\approx-\operatorname{sign}(x)\sigma(C(|x|-\Delta))$, with $\Delta=\operatorname{softplus}(\alpha)+\epsilon$ to enforce positivity. The useful property is a controllable dead zone with signed outlier preservation, while the smooth derivative lets thresholds and upstream weights receive gradients.

## Key formulas

- $$x_{\mathrm{sparse}}=\operatorname{sign}(x)\cdot\operatorname{ReLU}(|x|-\Delta)$$
- $$\left(\frac{dx_{\mathrm{sparse}}}{dx}\right)_{\mathrm{smooth}}=\sigma\big(C(|x|-\Delta)\big),\qquad \sigma(u)=\frac{1}{1+e^{-u}}$$
- $$\Delta=\operatorname{softplus}(\alpha)+\epsilon,\qquad \frac{\partial x_{\mathrm{sparse}}}{\partial\Delta}\approx-\operatorname{sign}(x)\sigma\big(C(|x|-\Delta)\big)$$
- $$\rho=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{|x_i|\leq\Delta\}$$

## Implementation notes

1. Integration point: start with the output activations of every large linear projection in an SSM block, including input, state, and output projections, or with transformer MLP and QKV output projections. Keep matrix weights unchanged initially; apply the gate immediately after the quantized matrix multiplication and before the next normalization or nonlinear operation. Store one trainable scalar $\alpha_j$ for each projection $j$; a second experiment can use one $\alpha_{j,k}$ per output channel. 2. Forward and backward pseudocode: compute $z=\operatorname{QuantLinear}_j(h)$; set $\Delta=\operatorname{softplus}(\alpha_j)+10^{-6}$; compute $y=\operatorname{sign}(z)\max(|z|-\Delta,0)$; during backward, replace the derivative with $g_z=\sigma(C(|z|-\Delta))$ and use $g_\Delta=-\operatorname{sign}(z)g_z$ for threshold learning. Implement this as a custom autograd function or straight-through estimator. Optionally add $\lambda\rho$ to the task loss, where $\rho$ is the fraction of values gated to zero, but first train without this term to measure natural sparsity. 3. Mathematics versus empirical estimates: the gate and surrogate derivatives come directly from the formulas above; $C$, threshold initialization, whether thresholds are per-projection or per-channel, and the sparsity penalty $\lambda$ are hyperparameters. Estimate $\rho$ on every validation batch and report it separately for every projection. 4. Cheap experiment: train a 100M-parameter S4/Mamba-style language model or small decoder-only transformer on WikiText-103 or C4 for the same number of tokens, comparing dense, fixed-threshold, and trainable-threshold gates. Initialize $\Delta$ at the activation 20th percentile so the initial zero rate is about 20%, then sweep target rates of 30%, 50%, and 75%. Measure validation perplexity, training loss at equal FLOPs, wall-clock time with a sparse kernel, fraction of zero activations, and energy if event-driven hardware is available. Success means at least 2x fewer nonzero activation events with less than 1% relative perplexity degradation, or faster loss descent at equal effective arithmetic operations.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.