Probability-coordinate robust likelihood head
Implementation & benchmark of arXiv:2609.03208 — G-Exponential Families through Probability Coordinates
Source paper: G-Exponential Families through Probability Coordinates arXiv:2609.03208 ⓘ · analyzed Sep 4, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace a Gaussian regression likelihood with a chart-pulled exponential-family head. The network predicts a natural parameter eta, while each target y is mapped to a bounded probability coordinate u=G(y). Because the sufficient statistic T(u) is bounded, extreme targets cannot create arbitrarily large parameter gradients. The chart determines the tail class, while eta controls the distribution in probability space.
Formulas
Mathematical statement
Let G:R -> (0,1) be a differentiable probability chart, let g(x)=G'(x), and let U=G(X). Start with the ordinary exponential-family density p_eta(u)=h(u) exp(eta T(u)-A(eta)) on (0,1), where h is a base density, T is a bounded sufficient statistic, eta is the neural-network output, and A(eta)=log integral from 0 to 1 of h(u) exp(eta T(u)) du. Pulling this family back to value space gives f_eta(x)=p_eta(G(x))g(x). For an observation y, the eta-dependent log-likelihood score is d log f_eta(y)/d eta = T(G(y))-A'(eta). Since T is bounded, this score is bounded up to the bounded range of A'(eta). The Fisher information is I(eta)=A''(eta)=Var_p_eta[T(U)], and the probability barycenter is b_G(f_eta)=G^{-1}(A'(eta)). Use the Cauchy chart G(x)=1/2+arctan(x)/pi and g(x)=1/(pi(1+x^2)) to obtain Cauchy-equivalent tails. A simple bounded statistic is T(u)=u-1/2, with range [-1/2,1/2].
Implementation notes
Integrate this as the final scalar probabilistic head of a small MLP or transformer regression model. The backbone produces a scalar z, and use eta=eta_max*tanh(z), with eta_max initially set to 8 to keep quadrature and gradients numerically safe. Use the Cauchy chart u=0.5+atan(y)/pi and statistic t=u-0.5. Choose h(u)=1 on (0,1), so A(eta)=log integral of exp(eta*(u-0.5)) du and A'(eta) is the corresponding mean statistic. For an MVP, precompute A and A' on a grid of 4096 eta values over [-8,8] using Gauss-Legendre quadrature with 64 or 128 nodes, then linearly interpolate during training. For each batch, execute: z=backbone(x); eta=eta_max*tanh(z); u=0.5+atan(y)/pi; t=u-0.5; A=interp_A(eta); nll=-eta*t+A-log(1/(pi*(1+y*y))); loss=mean(nll). The Jacobian term is necessary for calibrated value-space likelihoods but has no gradient with respect to eta. The formulas provide the chart, score, Fisher information, and barycenter; estimate quadrature tables numerically and measure gradient bounds empirically. First compare against Gaussian NLL, Huber regression, and Student-t NLL on y=x+epsilon with Cauchy noise and with 5 percent extreme outliers. Use a two-layer MLP and equal training budgets. Record per-example gradient magnitude versus |y|, optimization divergence rate, clean-test MAE, test NLL, and 90/95 percent interval coverage. Success is a bounded or saturating score for extreme targets, fewer unstable updates, improved contaminated-data accuracy, and better calibrated intervals at comparable compute.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.