Sampled Goldstein optimizer

Integrate this as an optimizer replacement at the parameter-update level, first for a small ReLU MLP or CNN. At each update, save the current parameter vector theta and draw m perturbations u_i uniformly from the sphere or normalized Gaussian directions with ||u_i||_2 = delta; evaluate the same minibatch loss at theta + u_i and backpropagate each loss to obtain g_i. Solve the simplex quadratic program min_alpha ||G alpha||_2^2 subject to alpha_i >= 0 and sum_i alpha_i = 1, where G has columns g_i. For a cheap MVP, use projected gradient descent on alpha or Frank-Wolfe: initialize alpha uniformly, repeatedly choose j = argmin_j <G alpha, g_j>, move alpha toward e_j, and stop after 10-20 iterations. Form g_star = G alpha, then apply the normalized update theta <- theta - eta*g_star/max(||g_star||_2,tau), with tau=1e-8. The paper's mathematical object is the sampled convex hull and its minimum-norm element; empirical choices are m, perturbation distribution, eta, and QP tolerance. Start with m=4 or 8, delta=epsilon, and reduce both by 10x after a fixed patience interval or when ||g_star|| falls below epsilon; reset momentum or optimizer state at each restart. Compare against AdamW and SGD on a 2-layer ReLU MLP trained on MNIST or a small CNN on CIFAR-10, using matched total gradient evaluations rather than matched optimizer steps. Measure loss versus gradient evaluations, gradient-direction variance across minibatches, final accuracy, and sensitivity to initialization. Success means lower loss or fewer activation-boundary instabilities at equal gradient evaluations; a meaningful negative result is no improvement after accounting for the m-fold gradient cost.