Gauge-fixed skew optimizer with exact norm conservation

(1) Integration point: modify the optimizer update for one or more weight tensors, preferably recurrent matrices, state-space transition matrices, or residual blocks whose norm stability matters. Flatten a weight tensor into $\theta$; use $Q=I$ for the MVP and optionally learn a scalar gain $\alpha$ with $W=\alpha\,\theta/\|\theta\|$ so the direction has fixed energy but the layer retains scale flexibility. Compute the ordinary minibatch gradient $g=\nabla_\theta\ell$.

(2) Pseudocode:

for minibatch:
    loss = model(x, y)
    g = grad(loss, theta)
    f = theta
    denom = dot(f, f) + eps
    v = -g + f * dot(f, g) / denom
    A = (outer(v, f) - outer(f, v)) / denom
    theta = solve(I - 0.5*h*A, (I + 0.5*h*A) @ theta)

For large layers do not materialize $A$: it is rank two, so solve the corresponding two-dimensional Krylov subspace exactly, or use a low-rank Woodbury solve. Recompute $A$ every optimizer step, or freeze it for several steps as an ablation. Use AdamW on all other parameters and compare against SGD, AdamW, and projected-gradient normalization.

(3) Computed from the paper's mathematics: $f$, the tangent projection $v$, the skew rank-two action, and the Cayley map. Estimated empirically: the step size $h$, regularizer $\varepsilon$, update frequency, and whether a learned gain is needed. Log $|\|\theta_{k+1}\|^2-\|\theta_k\|^2|$, gradient norms, activation norms, and training loss.

(4) First experiment: train a 6- to 12-layer residual MLP and a small GRU on CIFAR-10 or sequential MNIST, comparing AdamW, weight normalization, projected gradient, and the skew optimizer at matched parameter count and approximate FLOPs. The initial success signal is bounded parameter and activation norms and fewer optimization instabilities at large learning rates; a stronger result is equal or better validation accuracy with lower variance across seeds and no measurable drift in the conserved norm.