Hyperspatial Replica Optimizer

Integrate this at the optimizer level for a small MLP or Transformer block, not in the forward architecture exposed at evaluation. Choose a parameter subset such as the final MLP block, vectorize it as $\theta$, allocate a low-dimensional auxiliary vector $u\in\mathbb{R}^{q}$ with $q=8$ or $16$, and choose a fixed random orthonormal matrix $B$ mapping $u$ into that parameter subset. Maintain $R=4$ replicas with penalties $\mu_r$ logarithmically spaced, for example $[10^{-3},10^{-2},10^{-1},1]$. For each minibatch: (1) compute effective parameters $\theta^{(r)}_{\rm eff}=\theta^{(r)}+\epsilon Bu^{(r)}$; (2) evaluate loss $L_r$ and gradients of $L_r+\mu_r\|u^{(r)}\|^2/2$; (3) update $\theta^{(r)}$ and $u^{(r)}$ with AdamW or SGD; (4) every 50 steps, propose swaps between neighboring penalties, recompute the four augmented losses using the same minibatch, and accept with $a_{r,s}$ above. At checkpoint or inference time use the highest-penalty replica, set $u=0$, and evaluate the ordinary parameters $\theta$. The paper supplies the lifted-energy construction and confinement principle; $B$, $\epsilon$, the optimizer, and the swap schedule are engineering choices. Measure swap acceptance, auxiliary norm $\|u\|$, gradient variance, and the number of steps spent above a chosen loss threshold. First run CIFAR-10 with a 2-layer width-256 MLP and compare ordinary AdamW, four independent AdamW runs, and this method at equal total forward/backward FLOPs. A useful signal is faster escape from deliberately created barriers: initialize near a saturated ReLU regime or train on a two-cluster classification task with conflicting label phases, then test whether the method reaches the same final loss with fewer updates and lower variance. Also test a standard ResNet-18 on CIFAR-10 to determine whether any benefit survives realistic optimization rather than only synthetic barriers.