Post-Fixing Orthogonality Regularizer

Integrate the regularizer into the loss of a representation model, predictor, or graph neural network. Let each minibatch contain observed variables (x_i, y_i, z_i) and variables in the fixed set R. First compute post-fixing weights w_i. If the graph factorization gives the fixing density explicitly, evaluate w_R(o_i); otherwise estimate the density ratio between post-fixing and observational samples with a logistic classifier, clip weights to a fixed interval such as [0.1, 10], and renormalize them to have batch mean one. Choose test functions f_a and g_b as fixed random Fourier features, coordinate projections of the learned representation, or small auxiliary MLP critics. For each f_a, estimate r_a(z_i) = E_phi_R[f_a(X,Z) | Z = z_i] with an auxiliary regression network trained on the current batch or an exponential-moving-average buffer. Detach this regression prediction during the main update to reduce degenerate solutions. Compute the weighted moments m_hat_ab = sum_i w_i (f_a(x_i,z_i) - r_hat_a(z_i)) g_b(y_i,z_i) / sum_i w_i, then add lambda times the sum of squared moments to the task loss. Pseudocode: sample batch; compute or estimate fixing weights; compute f and g features; predict residual baselines r_hat; form weighted residual products; average over the batch; square and sum; add the penalty and weight-normalization term; backpropagate through the main model. The graph-derived weighting and residualization are mathematical inputs; weight clipping, critic capacity, auxiliary-regressor training, and lambda are empirical choices. Start with a small synthetic latent-confounding DAG for which the nested constraint is exactly known. Train an MLP baseline and the regularized model on observational data, then evaluate prediction risk and held-out weighted moment violations under interventions or distribution shifts. The first success signal is a lower held-out constraint violation at equal in-distribution accuracy, followed by improved out-of-distribution or interventional performance without increasing model size.