Polynomial anti-collapse regularizer

1. Integration point: attach the regularizer to an intermediate hidden tensor h with shape [batch, tokens, hidden_dim], such as the output of a Transformer MLP or the penultimate layer of an MLP classifier. Flatten batch and tokens, then randomly project h to k=8 coordinates so the polynomial feature dimension remains manageable. 2. Pseudocode: h = block(x); q = random_projection(h, hidden_dim, k); m = mean(q, axis=0).detach(); s = std(q, axis=0).detach(); z = (q-m)/(s+eps); phi = concat([ones, z, upper_triangle_products(z)]); c = phi-mean(phi,axis=0); S = transpose(c)@c/N; poly_loss = -slogdet(S+ridge*I); loss = task_loss + lambda_poly*poly_loss. Here the degree is d=2 and the lift contains all degree-0, degree-1, and degree-2 monomials, corresponding to a practical truncation of φ_r. 3. Compute the empirical covariance exactly as above; estimate the paper-inspired small-ball statistic separately by evaluating the maximum fraction of representations within radius t of a minibatch center. Treat the projection, ridge, regularizer weight, and truncation as empirical choices. Stop gradients through m and s initially, and use a small ridge such as 1e-4. 4. First experiment: train a 6-layer MLP and a small ViT on CIFAR-10, comparing standard LayerNorm, raw covariance log-determinant regularization, and the polynomial version at matched parameter count and FLOPs. Record validation accuracy, corruption accuracy, training-loss variance, smallest eigenvalue and log-determinant of S, and empirical small-ball values over several radii. Success is improved validation or corruption accuracy with fewer collapsed runs and no significant throughput loss.