Spectral-Margin Loop Regularizer

Add the regularizer to a tied recurrent transformer during training, using the Jacobian of one or a few sampled loop states rather than unrolling every state. At each optimizer step, select a minibatch and loop index t, flatten h_t, and define a Jacobian-vector product function v -> J_t v using automatic differentiation through the update h_next=A*h_t+Block(h_t+e)-(h_t+e). For small d or short sequences, explicitly materialize J_t and compute eigenvalues with eigvals; for realistic models use a blockwise approximation and 8-16 Arnoldi iterations to estimate the largest-magnitude eigenvalue. Set r=0.95 initially and add lambda_spec*[max(0,rho-r)]^2 to the task loss. To avoid severe transient explosions, optionally estimate ||J^k||_2 by power iteration on K=3 powers of the JVP and add the transient term with g=2 or 4. Backpropagate through the differentiable eigensolver when available; otherwise stop gradients through the estimated rho and use it as a controller that increases lambda_spec when rho exceeds r. The paper's mathematical quantity is rho(J); the empirical estimate is the Arnoldi or power-iteration approximation. First test a 4-loop, 2-layer tied transformer on a delayed-copy task and CIFAR-10, comparing no regularizer, operator-norm regularization, and spectral regularization at equal training FLOPs. Measure loss stability, gradient spikes, final accuracy, estimated rho, and transient amplification. The desired result is lower divergence frequency and better accuracy than norm control, while allowing some ||J||_2>1.