Boundary-Bifurcation Gradient Clipping

Integrate this at the optimizer step, initially with SGD or Adam plus global gradient-norm clipping. Treat the optimizer state, parameters, and optionally a low-dimensional projection of the parameters as the state \(y\). Use learning rate \(\lambda=\eta\) as the continuation parameter. At each checkpoint, record the unclipped norm \(r_t=\|g_t\|\), clipped fraction, loss, update norm, and an approximate stationarity residual \(q_t=\|\operatorname{EMA}(\Delta\theta_t)\|\). Pseudocode: compute \(g\); compute \(r=\|g\|\); set \(\tilde g=g\) if \(r\le C\), otherwise \(\tilde g=Cg/r\); update optimizer state and parameters; every K steps, repeat a short frozen-data probe at learning rates \(\eta-\delta\), \(\eta\), and \(\eta+\delta\). For each probe, run 10 to 50 deterministic steps with identical minibatches and estimate the local clipped branch by averaging the final state and clipped-gradient norm. Estimate \(s\) by finite difference of \(h=r-C\) along this branch: \(\hat s=[h(\eta+\delta)-h(\eta-\delta)]/(2\delta)\). The paper-derived structure is the switching surface and branch-slope criterion; the neural-network quantities are empirical estimates because exact equilibria and Jacobians are unavailable. First experiment: train a small MLP on MNIST and a two-layer Transformer on a tiny language corpus using SGD, fixed \(C\), and a learning-rate sweep. Compare the measured first clipping threshold and loss divergence against \(\hat s\). The quantitative prediction is that runs with \(\hat s\le0\) near first activation show an abrupt rise in loss variance or loss divergence within 50 steps, whereas runs with \(\hat s>0\) remain stable for at least 10 times longer. The transition should occur within 20 percent of the learning rate where the estimated branch slope changes sign.