Spherical Geometric-Gain Regularization

Integrate the method into the weight-regularization branch of a small MLP or Transformer feed-forward block, initially restricting it to square matrices so the paper's definition applies directly. At each training step, for every selected weight matrix W_l, draw K Gaussian vectors z_k in R^n, normalize them to u_k = z_k / (||z_k||_2 + delta), compute gains g_k = sqrt(||W_l u_k||_2^2 + epsilon^2), and compute h_l = exp(mean(log(g_k))). Add lambda_h times the sum of h_l to the task loss and backpropagate through these operations. Pseudocode is: sample z[K,n]; u = z / norm(z, axis=1); y = u @ W_l^T; h = exp(mean(log(sqrt(sum(y*y, axis=1) + epsilon^2)))); loss = task_loss + lambda_h * h; optimizer.step(). Use K=8 or 16 during training and resample directions every step; use K=256 at evaluation for a lower-variance diagnostic. The mathematical quantity from the paper is the exponential spherical log-average and its norm interpretation. The Monte Carlo estimator, epsilon, lambda_h, and direction reuse are engineering choices. Record h_l, its batch-to-batch variance, the spectral norm, and h_l / ||W_l||_2 to check whether the method differs from spectral regularization. The first cheap experiment is a three-layer width-256 MLP on MNIST or CIFAR-10, comparing no regularization, Frobenius regularization, spectral-norm regularization, and Hardy regularization at matched parameter count and comparable training loss. Evaluate validation accuracy, robustness to multiplicative input perturbations, layerwise activation and gradient growth, and largest singular values. A positive result is equal or better validation accuracy with reduced typical gain or improved depthwise stability, without the capacity loss caused by suppressing only the largest singular value.