Cosine-Covered Flat-Region Escape

Integrate this as a fallback branch after the normal AdamW update, operating independently on parameter blocks such as each transformer layer or each convolutional layer. At step t, compute the ordinary minibatch gradient g_t and AdamW proposal. If the gradient norm for a block is below a threshold tau_g, or if its exponential loss improvement over the last K steps is below tau_L, activate directional search. Sample m normalized Gaussian directions d_i in that block; for a cheap positively spanning pool, include antipodes so that S={d_1,-d_1,...,d_q,-d_q}, or use coordinate directions for very small blocks. Evaluate the same minibatch at theta+epsilon*d_i, preferably in one concatenated forward batch, and compute Delta_i=(L(theta+epsilon*d_i)-L(theta))/epsilon. Compute c=max_i[-Delta_i]_+/(max_i|Delta_i|+delta). If c is below c_min, resample directions once; if it remains low, skip the expensive escape step. Otherwise set weights lambda_i=softmax(-Delta_i/tau_d) over directions with Delta_i<0, form d=sum_i lambda_i d_i, and apply theta <- theta - eta*g + alpha*d, clipping alpha*d to a block norm proportional to the AdamW update norm. Do not backpropagate through the finite-difference evaluations. The paper-derived quantities are the positive-span construction and cosine-coverage criterion; epsilon, c_min, m, and alpha are empirical hyperparameters. Start with a 2-layer MLP on MNIST or CIFAR-10, compare AdamW against AdamW plus this fallback under matched total forward FLOPs, and test deliberately flat conditions such as saturated activations or a large learning-rate decay. Success is lower stall duration, more escapes from near-zero-gradient regions, and improved final accuracy at equal optimizer evaluations; a negative result is extra evaluations without faster loss descent or accuracy gain.