Variation-Diminishing Channel Mixer

Integrate the module immediately before the pointwise MLP in a small 1D CNN, an SSM block, or a transformer feed-forward block whose hidden channels have a fixed semantic or geometric order. Let \(h\in\mathbb{R}^{B\times L\times d}\) be the hidden tensor; apply the mixer independently along the ordered dimension \(d\), not along the batch or token dimensions. Replace a dense channel matrix by \(L_f\) bidiagonal factors. For each factor, use \(\alpha_\ell=\operatorname{softplus}(\theta_\ell)+10^{-4}\), compute \(z_1=\alpha_\ell h_1\), and for \(i>1\), compute \(z_i=h_{i-1}+\alpha_\ell h_i\); then set \(h\leftarrow z\). Optionally add a learned positive scalar gain after each factor and initialize all \(\alpha_\ell=1\). Pseudocode is: h=input; for theta in thetas: alpha=softplus(theta)+eps; z[...,0]=alpha*h[...,0]; z[...,1:]=h[...,:-1]+alpha*h[...,1:]; h=z; output=norm(h). The paper supplies the structural constraint and variation inequality; the engineer must estimate empirical sign variation using sign(h) after deleting entries with magnitude below \(10^{-6}\), and may estimate spectral norms by power iteration. First test a 4-layer MLP or 1D CNN on CIFAR-10 with ordered Fourier or DCT channels, comparing equal-parameter dense mixing, depthwise mixing, and the bidiagonal product at equal training steps and FLOPs. Pre-register: (1) on noisy high-frequency channel perturbations, the mixer must reduce median sign variation by at least 20% relative to its input and never increase it by more than 5%; (2) gradient norm coefficient of variation across depth should be at least 10% lower than the dense baseline; (3) clean accuracy must not fall by more than 1 percentage point. Measure variation before and after each mixer over 1,000 validation examples. Ablate only the positivity constraint by replacing \(\alpha\) with unconstrained parameters and allowing arbitrary subdiagonal weights. The transfer is falsified if sign variation routinely increases, if the constrained mixer loses over 2 points at matched compute, or if stability improvements disappear when the perturbation contains ordered-channel noise.