Finite Neumann Triangular Mixer

Use this as a structured mixer between m sequential Transformer blocks, MoE experts, or token groups. Store H as a tensor of shape B x m x T x d. Parameterize only blocks G_{ij} for i<j, using low-rank factors U_{ij}V_{ij}^T or a shared kernel G_{ij}=g(j-i)U_{j-i}V_{j-i}^T; set all i>=j blocks to zero. Apply optional gates M_i=sigmoid(g_i) or hard masks, then A=MGM. Compute Z without a matrix inverse using the recurrence S=H, P=H; for k=1,...,m-1, update P=-A(P), S=S+P; return Z=S. This is exactly the paper's finite Neumann formula, not a truncated approximation, provided the strict upper-triangular mask is preserved. Compute the algebraic residual r=mean(||Z+A Z-H||^2) during debugging. Estimate no spectral quantity: nilpotency follows structurally, while the empirical quantities are activation norm, gradient norm, and wall-clock cost. First test m=8 on a 12-layer small Transformer language model with d=256 and sequence length 512, replacing dense cross-layer adapters by the triangular mixer. Compare against a standard dense mixer and an iterative fixed-point inverse with the same parameter count. Report validation perplexity, peak activation memory, forward latency, and residual error. The expected signal is exact residual below numerical tolerance, fewer solver-related instabilities, and lower memory or latency when low-rank blocks are used, without worse perplexity at equal parameter count.