# Post-Fixing Orthogonality Regularizer

- ID: 139
- Canonical URL: https://synthcore.org/idea/139/post-fixing-orthogonality-regularizer
- API JSON: https://synthcore.org/api/idea/139.json
- API Markdown: https://synthcore.org/api/idea/139.md
- Verification status: unverified
- Source: [arXiv:2608.24602](https://arxiv.org/abs/2608.24602)
- Category: regularization
- Solves: accuracy, generalization, sample-efficiency
- ML areas: graph-nn, loss, regularization, representation
- Math tags: probability, statistics, geometry
- Ratings: usefulness 5/10; difficulty 6/10; novelty 5/10

## Idea description

Add a graph-derived conditional moment penalty to a neural representation or predictor. For each nested Markov constraint represented after fixing variables in R, residualize functions of (X,Z) with respect to Z under the post-fixing distribution and penalize their weighted correlation with functions of (Y,Z). This directly targets the equality constraint and can be more informative than an unconditional decorrelation penalty.

## Mathematical statement

The paper uses a post-fixing law P_phi_R, where phi_R is the graphical fixing operation on fixed-variable set R. Under the conditional independence X independent of Y given Z in this law, for any square-integrable functions f(X,Z) and g(Y,Z), Proposition 1 establishes the orthogonality relation E_phi_R[(f(X,Z) - E_phi_R[f(X,Z) | Z]) g(Y,Z)] = 0. Here X, Y, and Z are disjoint variable collections, f and g are test functions, and the residual f - E[f | Z] removes the component predictable from Z. If samples come from the observational law P rather than P_phi_R, represent the post-fixing expectation with a nonnegative density-ratio weight w_R(O) = dP_phi_R/dP(O), normalized so that E_P[w_R] = 1. A neural implementation minimizes the squared empirical weighted moments over several test functions f_a and g_b. Population moments vanish when the nested conditional-independence constraint holds.

## Key formulas

- $$X \perp Y\mid Z\ \text{under }{\mathbb P}_{\phi_R}\quad\Longrightarrow\quad {\mathbb E}_{\phi_R}[f(X,Z)\mid Y,Z]={\mathbb E}_{\phi_R}[f(X,Z)\mid Z].$$
- $${\mathbb E}_{\phi_R}\!\left[\left\{f(X,Z)-{\mathbb E}_{\phi_R}[f(X,Z)\mid Z]\right\}g(Y,Z)\right]=0.$$
- $$\widehat m_{ab}=\frac{1}{\sum_{i=1}^n w_i}\sum_{i=1}^n w_i\left(f_a(x_i,z_i)-\widehat r_a(z_i)\right)g_b(y_i,z_i),\qquad \widehat r_a(z_i)\approx {\mathbb E}_{\phi_R}[f_a(X,Z)\mid Z=z_i].$$
- $$\mathcal L_{\mathrm{total}}=\mathcal L_{\mathrm{task}}+\lambda\sum_{a=1}^{A}\sum_{b=1}^{B}\widehat m_{ab}^{\,2}+\eta\left(\frac{1}{n}\sum_i w_i-1\right)^2.$$

## Implementation notes

Integrate the regularizer into the loss of a representation model, predictor, or graph neural network. Let each minibatch contain observed variables (x_i, y_i, z_i) and variables in the fixed set R. First compute post-fixing weights w_i. If the graph factorization gives the fixing density explicitly, evaluate w_R(o_i); otherwise estimate the density ratio between post-fixing and observational samples with a logistic classifier, clip weights to a fixed interval such as [0.1, 10], and renormalize them to have batch mean one. Choose test functions f_a and g_b as fixed random Fourier features, coordinate projections of the learned representation, or small auxiliary MLP critics. For each f_a, estimate r_a(z_i) = E_phi_R[f_a(X,Z) | Z = z_i] with an auxiliary regression network trained on the current batch or an exponential-moving-average buffer. Detach this regression prediction during the main update to reduce degenerate solutions. Compute the weighted moments m_hat_ab = sum_i w_i (f_a(x_i,z_i) - r_hat_a(z_i)) g_b(y_i,z_i) / sum_i w_i, then add lambda times the sum of squared moments to the task loss. Pseudocode: sample batch; compute or estimate fixing weights; compute f and g features; predict residual baselines r_hat; form weighted residual products; average over the batch; square and sum; add the penalty and weight-normalization term; backpropagate through the main model. The graph-derived weighting and residualization are mathematical inputs; weight clipping, critic capacity, auxiliary-regressor training, and lambda are empirical choices. Start with a small synthetic latent-confounding DAG for which the nested constraint is exactly known. Train an MLP baseline and the regularized model on observational data, then evaluate prediction risk and held-out weighted moment violations under interventions or distribution shifts. The first success signal is a lower held-out constraint violation at equal in-distribution accuracy, followed by improved out-of-distribution or interventional performance without increasing model size.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.