Symplectic Recurrent Block

Insert the block between residual or state-space layers as a two-channel hidden state $(q,p)$, each with width $d$. Construct a scalar Hamiltonian $H_\phi(q,p,x)=U_\phi(q,x)+K_\phi(p)$, where $x$ is the block input, $U_\phi$ is a small MLP potential, and $K_\phi(p)=\frac12p^\top\operatorname{softplus}(m_\phi)^{-1}p$ with positive learned masses. Apply the three displayed leapfrog substeps, using automatic differentiation of the scalar Hamiltonian to obtain $\nabla_qH_\phi$ and $\nabla_pH_\phi$. Keep $x$ fixed during internal substeps. Use two or four substeps per block and expose either $q_{k+1}$ or a projected concatenation $[q_{k+1},p_{k+1}]$ to the next layer. Initialize $q$ from the incoming residual and $p=0$ or a learned projection of $x$. Train with the task loss; an optional small term $\lambda|H_{k+1}-H_k|^2$ may be used only as an ablation. The symplectic property follows analytically from the splitting, while the Hamiltonian scale and stable step size are learned or tuned. Test a six-layer residual MLP and a small Transformer on sequential MNIST, comparing Euler residual, gated residual, and symplectic blocks at matched parameters. Repeat each model 100–200 times at inference to test rollout stability. Record hidden-state norms, $H_k$, gradient norms, accuracy, and wall-clock cost. Success is bounded state and energy drift, fewer exploding gradients, and stable accuracy at long rollout lengths without disproportionate compute.