Parity-Aware Projective Feature Layer

Use this in a feature layer whose weights are constrained to unit-norm directions, especially cosine MLPs, random-feature regressors, or attention key/query projections when inputs are directional. Store only \(M\) representative directions \(w_i\) and coefficients \(a_i\); generate the antipodal partner on the fly rather than storing a second independent neuron. Set \(s=+1\) for an even activation such as absolute value or a deliberately symmetrized activation, and \(s=-1\) for an odd activation such as tanh. For each minibatch, compute both dot products \(w_i^\top x\) and \(-w_i^\top x\), combine them using the paired formula, backpropagate through the representative parameters only, and renormalize \(w_i\leftarrow w_i/\max(\|w_i\|,\epsilon)\). Estimate the induced kernel empirically on a validation subset using \(\widehat K(x,x')=M^{-1}\sum_i\phi_i(x)\phi_i(x')\), and compare its spectrum and condition number with an untied layer. The first cheap experiment is Gaussian single-index regression with even and odd teachers, comparing an untied width-\(2M\) network, a tied width-\(M\) network, and a standard width-\(M\) network. Success means approximately half the feature parameters and lower kernel redundancy at matched test MSE, together with improved stability across random seeds; failure is no accuracy gain or a substantial increase in optimization time from evaluating paired activations.