Sparse Weighing Mixer

(1) Integration point: replace one or more square dense MLP projections, preferably the value-to-hidden or hidden-to-output projection in a Transformer block, with a sparse channel mixer. For a hidden width M divisible by a catalog matrix order m, construct B=diag(W_1,...,W_q), normalize A_0=B/sqrt(k), and apply z=A x. Test three variants: fixed A=A_0; trainable A with the same zero support as B; and trainable A with the support fixed plus an orthogonality penalty.

(2) Pseudocode:

# precomputation: obtain W_r in {0,+1,-1}^{m x m}, W_r W_r^T=k I_m
B = block_diag(W_1,...,W_q)
mask = (B != 0)
A = B / sqrt(k)
for minibatch (x, target):
    A_sparse = mask * A
    z = x @ A_sparse.T
    y = activation(z)
    loss = task_loss(y, target)
    if orthogonal_variant:
        loss += (lambda/2) * frobenius_norm(A_sparse.T @ A_sparse - I_M)^2
    backpropagate(loss)
    optimizer_step(A)

Use a sparse kernel that stores only support indices and signs; the initial fixed matrix needs only additions/subtractions, and the trainable version stores one scalar per nonzero. For the fixed variant, add a learned diagonal gain G after the mixer, z=G A_0 x, to provide inexpensive feature-wise adaptation. The paper supplies the support/sign construction, sparsity k, and normalization 1/sqrt(k); the implementation must verify the identity numerically and estimate activation variance, gradient norms, and task accuracy empirically. If no catalog is available, generate candidate k-subsets per column and retain sign assignments satisfying W^TW=kI.

(3) First cheap experiment: train a small Transformer on WikiText-2 or a CIFAR-10 MLP, replacing 25%, 50%, and 100% of dense square MLP projections with the sparse mixer. Compare a dense baseline, random sparse matrices at the same k, fixed weighing mixers, and trainable fixed-support mixers. Measure validation perplexity or accuracy at equal training steps, wall-clock time, parameter count, peak memory, and activation/gradient norm variance. Success means lower wall-clock and memory at comparable accuracy, faster or more stable early loss descent than random sparse matrices, and substantially less norm drift in the fixed or orthogonality-regularized variants.