Weak-type pairwise smoothness penalty

Integrate the penalty after the encoder or classifier representation h_theta, using pairs (x_i,y_ij) where y_ij is a stochastic augmentation, a nearby training example, or a small adversarial perturbation of x_i. For every pair compute rho_ij=||a(x_i)-a(y_ij)||_2 in augmentation-parameter or input space, F_ij=||h_theta(x_i)-h_theta(y_ij)||_2, and g_ij=F_ij/(rho_ij+epsilon)^(1+gamma). Estimate V_ij with a k-nearest-neighbor volume proxy, such as V_ij=(k/N)*rho_k(x_i)^d, where d is a chosen intrinsic-dimension estimate and rho_k(x_i) is the kth-neighbor radius; detach rho and V from backpropagation. Use p=2 and sweep gamma in {0.5,1,2}. Set w_ij=rho_ij^(gamma*p)/V_ij and normalize the weights so they sum to one. Sort g values in each minibatch and evaluate the empirical weak norm at thresholds equal to the sorted values: R=max_q g_(q)*(sum_{r:g_r>g_(q)} w_r)^(1/p). To avoid discontinuous gradients, use a differentiable approximation such as tau*logsumexp((g_(q)*(tail_weight_q)^(1/p))/tau) with tau initially 0.1 and annealed toward 0. Add lambda*R to cross-entropy. The paper determines the weighted weak-tail form; pair sampling, kNN volume estimation, epsilon stabilization, and soft maximum are empirical adaptations. First test CIFAR-10 with a small ResNet-18 and random crop/color-jitter pairs, comparing no regularizer, mean squared feature-difference regularization, and the weak-type penalty at matched training FLOPs. Record clean accuracy, CIFAR-10-C corruption accuracy, the 95th and 99th percentile of normalized feature sensitivity, gradient-norm variance, and training loss oscillation. Success is reduced high-quantile sensitivity and improved corruption accuracy without clean-accuracy loss or materially slower optimization.