Totally-positive bidiagonal mixer

(1) Integration point: use this module as a replacement for a dense $d\times d$ projection in a Transformer feed-forward block, a token mixer, or the value/output projection in attention. Start with channel mixing because the same matrix is applied independently to every batch item and token. Keep the surrounding nonlinearities and residual connection unchanged. The module can also be inserted as a learned preconditioner immediately before an MLP.

(2) Pseudocode:

input X in R^[batch, length, d]
parameters ell[1:K,d-1], u[1:K,d-1], s[d]
a = softplus(ell); b = softplus(u); diag = exp(s)
Y = X * diag
for k = 1..K:
    Y = Y + shift_right(Y, 1) * a[k]
for k = K..1:
    Y = Y + shift_left(Y, 1) * b[k]
return residual_scale * Y

Apply each bidiagonal operation along the channel dimension; each factor costs $O(d)$ and the whole mixer costs $O(Kd)$ per vector, versus $O(d^2)$ for a dense projection. Use a norm or log-diagonal clamp if numerical growth appears, for example clip $s_k$ to $[-3,3]$.

(3) Computed from the paper: use the barycentric formula for $H_{\mathcal F}$ at initialization, perform Neville elimination or ordinary LU decomposition to obtain positive bidiagonal coefficients, and initialize $M$ to a normalized version of $H_{\mathcal F}$ for small $d$. During training, the softplus/exponential parameterization enforces positive edge and diagonal weights; no determinant constraints need be imposed. Estimate total positivity empirically by sampling minors of orders two through four and checking their signs, and estimate expressivity by comparing approximation error to a dense matrix on random targets.

(4) First cheap experiment: train a 2-layer Transformer on CIFAR-10 flattened patches or Tiny Shakespeare with $d=128$, replacing the dense MLP input projection by the $K=8$ positive bidiagonal mixer. Compare against the same parameter-count-matched dense layer, a low-rank linear layer, and an unconstrained product of bidiagonal factors. Measure validation loss or accuracy, gradient-norm variance, singular-value growth, parameter count, and wall-clock/FLOPs. The initial success signal is equal or better validation accuracy with fewer parameters and lower gradient instability; for a sequence mixer, also test whether the positive factorization reaches the dense baseline at $K\ll d$.