Oja Gradient-Subspace Optimizer

1. Integration point: start with a small MLP or transformer block and maintain one Oja basis per weight matrix, or one basis for the concatenated gradients of a selected layer. Flatten the minibatch gradient to g_t, update the covariance estimate, update U_t, and use the projected or softly projected gradient in the ordinary optimizer. Do not materialize A_t for large layers; estimate A_t U_t directly as beta times the previous A_U plus (1-beta) times g_t times (g_t^T U_t), followed by the projected Oja increment. 2. Pseudocode: compute g; set A_U = beta A_U + (1-beta) g(g^T U); set Z = U + (h/epsilon)(A_U - U(U^T A_U)); set U = qr(Z); set P_g = U(U^T g); set g_tilde = P_g + alpha(g - P_g); apply Adam or SGD to theta using g_tilde. 3. Computed from the paper: the Oja vector field, Stiefel projection, and eigengap-dependent response mechanism. Estimated empirically: lambda_r, lambda_(r+1), subspace drift, and the useful value of epsilon. 4. First cheap experiment: train a width-512 MLP on CIFAR-10 with SGD, comparing SGD, low-rank projected SGD, and soft projection for r in {8, 32, 128}. Freeze covariance statistics after a stationary phase and measure principal-angle error against a periodically computed exact PCA basis. For a fixed covariance with measured gap Delta, vary h/epsilon and fit the error decay; the predicted slope is -Delta/epsilon in continuous time, or log(rho)/h approximately equal to -Delta/epsilon. Under a controlled rotating covariance whose top subspace rotates at angular speed omega, the tracking error should increase sharply when omega becomes comparable to Delta/epsilon; test whether the observed transition is within 20 percent of this boundary. Reject settings where the discrete increment loses orthogonality or causes exploding training loss.