Correlated-Gaussian Orbit Fingerprint

Integrate this at the output of a polynomial or tensorized MLP block. Represent the block as P_theta(x)=sum_{k=0}^m <T_k,x^{tensor k}>, or use a degree-m polynomial activation whose coefficients are produced by a small neural network. During training, create r replicas for each probe: sample Z_a independently from N(0,I_d), factor a chosen covariance Sigma_q=L_q L_q^T, and set X_a=sum_b (L_q)_{ab} Z_b. Evaluate P_theta(X_a) and accumulate the product over replicas; this Monte Carlo estimate is the tensor M_hat_{theta,r}(Sigma_q). Use a fixed bank of rational, well-conditioned Sigma_q satisfying 0.5 I <= Sigma_q <= 1.5 I, including identity, pairwise-correlated matrices, and random symmetric directions Sigma=I+epsilon A. For each minibatch, compute the fingerprint vector F_theta by concatenating M_hat over r=1,...,R and q=1,...,Q. Add lambda_fp times the squared distance between fingerprints of two orthogonally transformed views or two parameterizations known to represent the same function; alternatively use F_theta as an auxiliary embedding and apply a contrastive loss to separate teacher functions. Pseudocode: sample q,r; construct correlated X; y_a=P_theta(X_a); m_q=mean(batch,prod_a y_a); concatenate m_q; optimize task_loss+lambda_fp*contrastive_loss(F). The paper's exact mathematical computation is the Gaussian replica moment and covariance-polynomial degree bound. Estimate moments by Monte Carlo, and estimate interpolation coefficients only if desired using the Lagrange formula with K+1 covariance values along each affine direction. Start with a degree-3 tensor MLP on MNIST or CIFAR-10, compare against the same model with independent-replica moments or no fingerprint loss, and measure validation accuracy, robustness to random orthogonal reparameterizations, and whether functions with matched marginal P(X) become separable. A success signal is unchanged task loss but lower performance variance under orthogonal changes and improved accuracy when training data are limited.