Mixed-Type Conditional-Invariance Regularizer

(1) Integration point: apply this during training to a representation R_i=f_theta(X_i), while Y_i is a protected attribute, nuisance label, or future target whose information should be conditionally absent from R_i given context Z_i. Use a differentiable kernel such as kappa(y_i,y_j)=exp(-||e(y_i)-e(y_j)||^2/(2 sigma^2)) for continuous or learned target embeddings, and keep neighbor indices detached so the loss is piecewise differentiable through similarities. (2) Pseudocode: sample a batch; construct C_i by exact equality on discrete Z fields and k-nearest-neighbor distance on normalized continuous Z; construct F_i by exact matching on discrete (Z,R) fields where applicable and k-nearest-neighbor distance on concatenated (Z,R); compute s_i(C_i) and s_i(F_i), then d_i. For each i, select m nearby points in continuous Z, form polynomial features p(z)=[1,z_1,...,z_q] optionally with quadratic terms, use diagonal weights W_jj=exp(-||z_j-z_i||^2/h^2), solve the ridge weighted least-squares expression for b_hat_i, and minimize the squared mean residual plus the task loss. (3) Compute from the paper's construction: mixed-type neighborhood definitions, local similarity contrast, and local-polynomial bias correction. Estimate empirically: k, bandwidth h, kernel scale sigma, ridge lambda, and penalty weight alpha; tune them on validation data. (4) First experiment: train a small MLP on Adult or a synthetic mixed-type regression dataset to predict Y from X with categorical and continuous Z. Compare the baseline, ordinary adversarial conditional-invariance training, and this regularizer at equal batch size and FLOPs. Measure task accuracy, a held-out kNN conditional-independence statistic, and sensitive-variable prediction accuracy from R conditional on Z. Success is lower held-out conditional dependence at no more than 1% task-accuracy loss, with near-quadratic batch computation rather than a dense cubic global-kernel computation.