Blow-Up Annealing for Heterogeneous Sharpness

1. Integration point: choose an MLP or transformer feed-forward block with two gated branches, such as an attention branch and an MLP branch or two parallel MLP experts. Replace each sigmoid or GELU gate by a temperature-controlled sigmoid $H(z;\varepsilon,\tau)$, using independent temperatures $\varepsilon_1$ and $\varepsilon_2$. Parameterize them as $\varepsilon_1=r$ and $\varepsilon_2=r\rho$ in the first chart; use the swapped chart when $\rho>1$ if numerical conditioning becomes poor. 2. Pseudocode: initialize $r=r_0$ and $\rho=1$; every $K$ optimization steps estimate the largest eigenvalue magnitude of the update Jacobian using one or two Hessian-vector products and power iteration. If $\rho(J_t)>1-\delta$, stop decreasing $r$, reduce the learning rate by $0.5$, and move $\rho$ toward the less-sharp branch; otherwise set $r\leftarrow qr$ with $q\in[0.95,0.999]$. Log loss, gradient norm, $\rho(J_t)$, activation saturation, and the ratio $\rho$. 3. Computed from the paper: the directional blow-up construction and the prediction that different relative scalings have different limiting dynamics. Estimated empirically: the neural-network stability boundary, since exact bifurcation values depend on architecture, data, and optimizer. 4. First cheap experiment: train a two-branch three-layer MLP on MNIST or CIFAR-10 with SGD, comparing joint annealing $(\varepsilon_1,\varepsilon_2)=(r,r)$ against fixed-ratio schedules $\rho\in\{0.1,0.3,1,3,10\}$. Use identical final temperatures and learning-rate budgets. The falsifiable prediction is a sharp change in loss oscillation, gradient variance, or saturation when $\rho$ crosses an architecture-dependent interval; the stable schedule should maintain $\rho(J_t)<1$ and permit a smaller final $r$ without divergence. A quantitative target is that the measured transition in the dominant Jacobian eigenvalue occurs consistently within 20% across random seeds and at least two widths.