Unverified 2026

Probability-coordinate robust likelihood head

Implementation & benchmark of arXiv:2609.03208 — G-Exponential Families through Probability Coordinates

Usefulness6/10
Difficulty4/10
Novelty6/10

Source paper: G-Exponential Families through Probability Coordinates arXiv:2609.03208 · analyzed Sep 4, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a Gaussian regression likelihood with a chart-pulled exponential-family head. The network predicts a natural parameter eta, while each target y is mapped to a bounded probability coordinate u=G(y). Because the sufficient statistic T(u) is bounded, extreme targets cannot create arbitrarily large parameter gradients. The chart determines the tail class, while eta controls the distribution in probability space.

Formulas

$$f_{\eta}(x)=p_{\eta}(G(x))\,g(x),\qquad p_{\eta}(u)=h(u)\exp\bigl(\eta T(u)-A(\eta)\bigr),\qquad g(x)=G'(x).$$
$$A(\eta)=\log\int_0^1 h(u)\exp\bigl(\eta T(u)\bigr)\,du,\qquad \frac{\partial}{\partial\eta}\log f_{\eta}(y)=T(G(y))-A'(\eta).$$
$$b_G(f_{\eta})=G^{-1}\bigl(A'(\eta)\bigr),\qquad I(\eta)=A''(\eta)=\operatorname{Var}_{p_{\eta}}[T(U)].$$
$$G_{\mathrm{Cauchy}}(x)=\frac12+\frac{1}{\pi}\arctan(x),\qquad g_{\mathrm{Cauchy}}(x)=\frac{1}{\pi(1+x^2)},\qquad \ell(y;\eta)=-\eta T(G(y))+A(\eta)-\log g(y).$$

Mathematical statement

Let G:R -> (0,1) be a differentiable probability chart, let g(x)=G'(x), and let U=G(X). Start with the ordinary exponential-family density p_eta(u)=h(u) exp(eta T(u)-A(eta)) on (0,1), where h is a base density, T is a bounded sufficient statistic, eta is the neural-network output, and A(eta)=log integral from 0 to 1 of h(u) exp(eta T(u)) du. Pulling this family back to value space gives f_eta(x)=p_eta(G(x))g(x). For an observation y, the eta-dependent log-likelihood score is d log f_eta(y)/d eta = T(G(y))-A'(eta). Since T is bounded, this score is bounded up to the bounded range of A'(eta). The Fisher information is I(eta)=A''(eta)=Var_p_eta[T(U)], and the probability barycenter is b_G(f_eta)=G^{-1}(A'(eta)). Use the Cauchy chart G(x)=1/2+arctan(x)/pi and g(x)=1/(pi(1+x^2)) to obtain Cauchy-equivalent tails. A simple bounded statistic is T(u)=u-1/2, with range [-1/2,1/2].

Implementation notes

Integrate this as the final scalar probabilistic head of a small MLP or transformer regression model. The backbone produces a scalar z, and use eta=eta_max*tanh(z), with eta_max initially set to 8 to keep quadrature and gradients numerically safe. Use the Cauchy chart u=0.5+atan(y)/pi and statistic t=u-0.5. Choose h(u)=1 on (0,1), so A(eta)=log integral of exp(eta*(u-0.5)) du and A'(eta) is the corresponding mean statistic. For an MVP, precompute A and A' on a grid of 4096 eta values over [-8,8] using Gauss-Legendre quadrature with 64 or 128 nodes, then linearly interpolate during training. For each batch, execute: z=backbone(x); eta=eta_max*tanh(z); u=0.5+atan(y)/pi; t=u-0.5; A=interp_A(eta); nll=-eta*t+A-log(1/(pi*(1+y*y))); loss=mean(nll). The Jacobian term is necessary for calibrated value-space likelihoods but has no gradient with respect to eta. The formulas provide the chart, score, Fisher information, and barycenter; estimate quadrature tables numerically and measure gradient bounds empirically. First compare against Gaussian NLL, Huber regression, and Student-t NLL on y=x+epsilon with Cauchy noise and with 5 percent extreme outliers. Use a two-layer MLP and equal training budgets. Record per-example gradient magnitude versus |y|, optimization divergence rate, clean-test MAE, test NLL, and 90/95 percent interval coverage. Success is a bounded or saturating score for extreme targets, fewer unstable updates, improved contaminated-data accuracy, and better calibrated intervals at comparable compute.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.