Marchenko–Pastur Tensor Initialization

Integrate this at the input of a tensorized random-feature layer, polynomial-kernel approximation, or low-rank tensor MLP. (1) Given a minibatch x_i in R^q, construct u_i=vec(x_i^{tensor d}) using repeated Kronecker products, or use a fixed random orthogonal sketch if q^d is too large. Stack the rows into U in R^{B times p}, where p is the retained feature dimension. (2) Compute s_d^2=(1/B) sum_i ||u_i||_2^2, set z_i=u_i/s_d, and pass Z to the following linear layer. If the target covariance scale is identity, additionally divide by sqrt(p) only if the implementation represents rows as unnormalized random features. (3) Use c=p/m_eff, where m_eff is the number of independent examples represented by the covariance estimate. Compute a_c=(1-sqrt(c))^2 and b_c=(1+sqrt(c))^2 when c<=1. If c is near one, reduce p, increase feature accumulation, or use a sketch; if c>1, do not expect full-rank covariance. (4) Every several hundred steps, estimate the smallest and largest eigenvalues of Z^T Z/B with 5–10 power or Lanczos iterations and compare them with the predicted edges. The mathematical prediction is the MP edge formula; the actual finite-batch spectrum and any correction factor are empirical. First test q=32 and d=2 or 3 in a two-layer MLP on CIFAR-10 or synthetic teacher regression. Compare independent Gaussian, correlated exchangeable, and MP-scaled tensor features at equal p and FLOPs. Log activation variance, gradient norms, eigenvalue edges, training loss, and validation accuracy. Success is a larger minimum eigenvalue, fewer exploding or vanishing-gradient events, and faster loss descent at equal compute.