Spin-Wave Nonlinearity Damping

Integrate the gate at the residual-update coefficient of a Transformer block, initially in the MLP branch rather than attention. For hidden state h with shape [batch, sequence, dimension], compute z=LayerNorm(h), then u=F_theta(z), where F_theta is the usual gated MLP or SwiGLU. Maintain a scalar or per-channel variance EMA. The update is: (1) compute v_batch=mean((z-mean(z))^2) over batch and sequence dimensions; (2) update v_ema=beta*v_ema+(1-beta)*stop_gradient(v_batch); (3) compute g=exp(-0.5*alpha*clip(v_ema,0,v_max)); (4) return h_new=h+lambda*g*u. Initialize alpha in {0.1, 0.3, 1.0}, beta=0.99, and v_max=10. Use stop-gradient through the EMA in the first implementation, then test a differentiable variant. The paper-derived object is the exponential attenuation exp(-C0/2); the engineering approximation identifies C0 with observed activation fluctuation variance and alpha with an adjustable noise-to-diffusivity ratio. Log g, v_ema, gradient norms, and the fraction of gates below 0.1. Start with a 6-layer small Transformer on WikiText-2 or CIFAR-10 image tokens, comparing standard Pre-LN residuals, fixed residual scaling, RMSNorm, and this gate at matched parameter count and FLOPs. Stress-test with larger learning rates, injected embedding noise, and small batch sizes. Success means fewer loss spikes, lower maximum gradient norm, and equal or better validation loss at the same training budget; a strong result is stable training at a learning rate where the ungated baseline diverges.