Dual-gauge cross-stream block
Source paper: Gauge-compatible tensors on statistical manifolds: splitting and submanifold geometry arXiv:2608.31145 ⓘ · analyzed Sep 2, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace an unconstrained hidden-to-hidden interaction in an MLP or transformer feed-forward block by two gauge-related branches. Split channels with an orthogonal involution Θ, constrain the learned interaction K to anticommute with Θ, and use opposite signs of K in paired branches. This creates a testable inductive bias in which the learned interaction only transfers information between the two channel subspaces.
Formulas
Mathematical statement
A statistical manifold has a metric g and dual connections ∇ and ∇* satisfying Xg(Y,Z)=g(∇_X Y,Z)+g(Y,∇*_X Z). Relative to the Levi-Civita connection ∇ᵍ, the connections are ∇=∇ᵍ+K and ∇*=∇ᵍ−K, where K is the statistical difference tensor. The paper defines the gauge equation ∇_X(ΘY)=Θ(∇*_X Y). If Θ is parallel under ∇ᵍ, this reduces to K_X Θ=−Θ K_X. For a neural block, choose Θ as a fixed orthogonal involution with Θ²=I and ΘᵀΘ=I, for example Θ=diag(I₍d₊₎,−I₍d₋₎). The anticommutation law forces K to be off-diagonal: K=[[0,A],[B,0]], so the positive channels interact only with negative channels and vice versa. Let L be a base map commuting with Θ. The paired residual maps F₊(h)=h+Lh+Kh and F₋(h)=h+Lh−Kh then satisfy the discrete gauge identity F₊(Θh)=ΘF₋(h).
Implementation notes
(1) Integration point: replace one transformer FFN residual sublayer, or one hidden-to-hidden MLP block, with a dual-gauge block. For hidden width d=d₊+d₋, fix Θ=diag(+1,…,+1,−1,…,−1), usually with d₊=d₋=d/2. Apply RMSNorm or LayerNorm to input h. Compute a shared base map L(h), and compute an interaction K(h) using only cross-partition weights. If h=(h₊,h₋), define K(h)=(A h₋, B h₊), where A and B are learned matrices or two-layer gated MLPs. (2) Pseudocode: x=norm(h); base=block_diagonal_mlp(x); cross_plus=A(x_minus); cross_minus=B(x_plus); k=concat(cross_plus,cross_minus); y=h+alpha*(base+k). Define the dual branch with y_dual=h+alpha*(base-k). Use the first branch in the ordinary network, or average y with Theta*(dual(Theta*x)) for an explicitly symmetrized version. (3) The off-diagonal form is exact by parameterization; estimate the gauge residual as r=||F₊(Θx)−ΘF₋(x)||₂/(||x||₂+10⁻⁸). If L is not block diagonal, also measure c=||LΘ−ΘL||F/(||L||F+10⁻⁸). (4) First experiment: compare a standard six-layer width-256 transformer, a randomly split two-stream FFN with the same parameter count, and this gauge block on TinyStories language modeling or CIFAR-10 classification. Match parameters, training steps, FLOPs, optimizer, batch size, and learning-rate schedule, using at least three seeds. (5) Pre-register mechanism tests: r must remain below 10⁻⁴; removing the off-diagonal parameterization should increase r by at least 100 times. If the structure improves stability, gradient-norm spike frequency and validation-loss variance across seeds should decrease by at least 10% at equal compute. Test widths 128, 256, and 512 and report loss versus FLOPs. Falsify the transfer if the constrained model has no stability improvement, no accuracy or efficiency benefit, and no measurable relation between the gauge residual and training behavior.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.