{
 "artifacts": null,
 "category": "architecture",
 "description": "Replace the dense output of selected linear projections with a two-sided magnitude threshold that emits zero for small values but preserves signed large values. Learn one positive threshold per projection, or optionally one threshold per output channel, so the network discovers where sparse events can be removed while retaining outlier information.",
 "formulas_latex": [
  "$$x_{\\mathrm{sparse}}=\\operatorname{sign}(x)\\cdot\\operatorname{ReLU}(|x|-\\Delta)$$",
  "$$\\left(\\frac{dx_{\\mathrm{sparse}}}{dx}\\right)_{\\mathrm{smooth}}=\\sigma\\big(C(|x|-\\Delta)\\big),\\qquad \\sigma(u)=\\frac{1}{1+e^{-u}}$$",
  "$$\\Delta=\\operatorname{softplus}(\\alpha)+\\epsilon,\\qquad \\frac{\\partial x_{\\mathrm{sparse}}}{\\partial\\Delta}\\approx-\\operatorname{sign}(x)\\sigma\\big(C(|x|-\\Delta)\\big)$$",
  "$$\\rho=\\frac{1}{N}\\sum_{i=1}^{N}\\mathbf{1}\\{|x_i|\\leq\\Delta\\}$$"
 ],
 "id": 2889,
 "implementation": "1. Integration point: start with the output activations of every large linear projection in an SSM block, including input, state, and output projections, or with transformer MLP and QKV output projections. Keep matrix weights unchanged initially; apply the gate immediately after the quantized matrix multiplication and before the next normalization or nonlinear operation. Store one trainable scalar $\\alpha_j$ for each projection $j$; a second experiment can use one $\\alpha_{j,k}$ per output channel. 2. Forward and backward pseudocode: compute $z=\\operatorname{QuantLinear}_j(h)$; set $\\Delta=\\operatorname{softplus}(\\alpha_j)+10^{-6}$; compute $y=\\operatorname{sign}(z)\\max(|z|-\\Delta,0)$; during backward, replace the derivative with $g_z=\\sigma(C(|z|-\\Delta))$ and use $g_\\Delta=-\\operatorname{sign}(z)g_z$ for threshold learning. Implement this as a custom autograd function or straight-through estimator. Optionally add $\\lambda\\rho$ to the task loss, where $\\rho$ is the fraction of values gated to zero, but first train without this term to measure natural sparsity. 3. Mathematics versus empirical estimates: the gate and surrogate derivatives come directly from the formulas above; $C$, threshold initialization, whether thresholds are per-projection or per-channel, and the sparsity penalty $\\lambda$ are hyperparameters. Estimate $\\rho$ on every validation batch and report it separately for every projection. 4. Cheap experiment: train a 100M-parameter S4/Mamba-style language model or small decoder-only transformer on WikiText-103 or C4 for the same number of tokens, comparing dense, fixed-threshold, and trainable-threshold gates. Initialize $\\Delta$ at the activation 20th percentile so the initial zero rate is about 20%, then sweep target rates of 30%, 50%, and 75%. Measure validation perplexity, training loss at equal FLOPs, wall-clock time with a sparse kernel, fraction of zero activations, and energy if event-driven hardware is available. Success means at least 2x fewer nonzero activation events with less than 1% relative perplexity degradation, or faster loss descent at equal effective arithmetic operations.",
 "math_summary": "The paper's sparse pre-activation is $x_{\\mathrm{sparse}}=\\operatorname{sign}(x)\\operatorname{ReLU}(|x|-\\Delta)$, where $x\\in\\mathbb{R}$ is a scalar pre-activation and $\\Delta\u003e0$ is a trainable threshold associated with a linear projection. Equivalently, $x_{\\mathrm{sparse}}=0$ when $|x|\\leq\\Delta$, $x-\\Delta$ when $x\u003e\\Delta$, and $x+\\Delta$ when $x\u003c -\\Delta$; therefore both positive and negative outliers survive. The paper avoids completely zero gradients at the threshold using the surrogate derivative $\\left(\\frac{dx_{\\mathrm{sparse}}}{dx}\\right)_{\\mathrm{smooth}}=\\sigma(C(|x|-\\Delta))$, where $\\sigma(u)=1/(1+e^{-u})$ is the logistic sigmoid and $C\u003e0$ controls transition sharpness. For a trainable threshold, use the corresponding surrogate $\\partial x_{\\mathrm{sparse}}/\\partial\\Delta\\approx-\\operatorname{sign}(x)\\sigma(C(|x|-\\Delta))$, with $\\Delta=\\operatorname{softplus}(\\alpha)+\\epsilon$ to enforce positivity. The useful property is a controllable dead zone with signed outlier preservation, while the smooth derivative lets thresholds and upstream weights receive gradients.",
 "math_tags": [
  "optimization",
  "approximation-theory",
  "dynamical-systems"
 ],
 "ml_areas": [
  "ssm",
  "transformer",
  "inference-speedup",
  "quantization"
 ],
 "paper": {
  "arxiv_id": "2608.30439",
  "arxiv_url": "https://arxiv.org/abs/2608.30439",
  "summary_what_math_gives_to_ml": "The paper provides a simple, hardware-relevant sparsification operator rather than merely pruning weights: it removes low-magnitude activation events while retaining both positive and negative outliers. The transferable asset is the trainable, per-projection dead zone, which can convert dense matrix operations into event-driven sparse computation without collapsing the activation distribution to only positive values. A practical adaptation is to insert this operator after quantized linear projections in an SSM or transformer and train its thresholds with a smooth surrogate gradient, explicitly measuring the tradeoff between activation sparsity and task loss. The main uncertainty is whether GPU implementations benefit, but the method is directly testable for neuromorphic hardware, sparse kernels, and activation-aware inference.",
  "title": "Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 5,
  "usefulness": 6
 },
 "solves": [
  "speedup",
  "memory",
  "accuracy"
 ],
 "title": "Trainable two-sided activation gate",
 "url": "https://synthcore.org/idea/2889/trainable-two-sided-activation-gate",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}