Sparse Root-of-Unity Isometric Mixer

(1) Integration point: replace one dense channel-mixing operation in an MLP block, residual block, or Transformer feed-forward projection by y = U x, where U is fixed during the first experiment. Use k=2 for a real-valued MVP. Store each row as w integer column indices and w signs, so the operation is a sparse signed gather-and-sum followed by multiplication by 1 divided by square root of w. For a complex k=4 version, represent each complex feature as two real channels and implement phases using the four maps {x, -x, i x, -i x}, which require only sign changes and real-imaginary swaps. (2) Pseudocode: generate candidate W; set every row and column to exactly w nonzeros; assign entries from the chosen root-of-unity table; compute G = W W*; reject the candidate unless max absolute off-diagonal(G) is below 1e-6 and max absolute diagonal(G)-w is below 1e-6. During forward propagation, for each row i compute y_i = sum over r from 1 to w of phase[i,r] times x[idx[i,r]] divided by sqrt(w). Apply the original bias and activation after this mixer. (3) Mathematical checks are the exact weighing identity and the resulting norm preservation ||Ux||_2 = ||x||_2. If reproducing the paper's structured construction, enumerate a small finite group G, subgroup H, transversal T, and character chi; assign edge patterns using the centralizer equation and discard non-orientable orbitals using the commutator condition above. Empirically measure activation-norm ratios, backpropagated gradient-norm ratios, sparse-kernel wall-clock time, and validation loss. (4) First cheap experiment: train a six-layer width-512 residual MLP on CIFAR-10, replacing every second dense mixer with a CGW mixer at w in {16,32,64}. Compare against the dense baseline and a random signed sparse matrix with identical sparsity. Keep optimizer, initialization, and parameter count otherwise fixed. Record accuracy, loss versus optimizer step, gradient-norm variance, mixer FLOPs, parameter memory, and actual examples per second. The idea succeeds if it matches or exceeds random sparse accuracy, remains near-isometric across depth, and provides lower measured mixing cost; a concrete target is at least 2 times fewer mixing multiply-adds without an accuracy drop larger than 0.5 percentage points.