Relative-Noise Loss for Covariance Ratios

Integrate this into a neural causal estimator or representation-learning model whose encoder produces samples \(z_X,z_M,z_Y\), and whose prediction head computes a front-door-style ratio \(r_\theta=N_\theta/D_\theta\) from minibatch covariances. Do not regularize \(D_\theta\) toward a fixed absolute magnitude. First compute centered minibatch covariance entries \(\hat\sigma_{UV}=B^{-1}\sum_{i=1}^B(z_{U,i}-\bar z_U)(z_{V,i}-\bar z_V)\). Parameterize any predicted population covariance as \(\Sigma_\theta=L_\theta L_\theta^\top+\delta I\), with positive diagonal entries in \(L_\theta\), then form \(D_\theta=\sigma_{XX,\theta}(\sigma_{XX,\theta}\sigma_{MM,\theta}-\sigma_{XM,\theta}^2)\). For each batch, compute \(\hat D\) from the empirical covariance and add the paper-derived relative loss \(\mathcal L_{rel}=(B/10)[\log(\hat D+\varepsilon)-\log(D_\theta+\varepsilon)]^2\) to the ordinary outcome loss. If a ratio target \(r\) is available, use the same scale principle for its denominator-related uncertainty: multiply denominator residuals by \(1/(D_\theta+\varepsilon)\), not by a fixed raw-scale coefficient. The factor 10 and the \(B\)-scaling come from the Gaussian asymptotic result; covariance entries, \(D_\theta\), and the empirical loss are estimated by minibatch statistics. Start with a three-layer MLP on synthetic linear front-door data generated from Gaussian structural equations, comparing ordinary MSE plus raw-denominator penalty against the proposed loss. Sweep mediator residual variance over several orders of magnitude while keeping the treatment–mediator coefficient nonzero. Success means stable ratio predictions and nominal bootstrap coverage or lower test MSE when the denominator is numerically small, without systematic rejection of valid low-variance representations; also verify that gradient norms and training loss do not explode as \(D\) shrinks.