# Entropic Projection Mixture Head

- ID: 327
- Canonical URL: https://synthcore.org/idea/327/entropic-projection-mixture-head
- API JSON: https://synthcore.org/api/idea/327.json
- API Markdown: https://synthcore.org/api/idea/327.md
- Verification status: unverified
- Source: [arXiv:2607.01229](https://arxiv.org/abs/2607.01229)
- Category: architecture
- Solves: accuracy, generalization, stability
- ML areas: rl, loss, world-model, embedding
- Math tags: probability, convex-analysis, statistics, optimization
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Replace a single scalarization of a vector-valued stochastic prediction with a positive mixture of entropic certainty equivalents applied to positive projections. The head can represent both coordinate priorities and risk sensitivity, and it distinguishes correlated or complementary outcomes that receive the same value under a fixed linear average.

## Mathematical statement

For a vector risk X=(X_1,...,X_d) with X taking values in V=\mathbb{R}^d, a shadow valuation is the scalar projection q\cdot X=\sum_{j=1}^d q_jX_j, where q is a nonnegative weight vector in the positive dual cone q\in\mathbb{R}_+^d. The paper's representation says that an admissible multidimensional certainty equivalent can be represented as a positive mixture of scalar entropic certainty equivalents of these projections. The scalar entropic certainty equivalent at risk-aversion parameter \eta>0 is CE_\eta(Z)=-\eta^{-1}\log\mathbb{E}[\exp(-\eta Z)]. The neural adaptation uses a probability measure \mu over positive projection/risk-aversion pairs (q,\eta): C(X)=\mathbb{E}_{(q,\eta)\sim\mu}[ -\eta^{-1}\log\mathbb{E}[\exp(-\eta q\cdot X)] ]. The extracted text gives the concrete failure mode of fixed linear scalarization: with X=(1,0) or (0,1), each with probability 1/2, and Y=(1/2,1/2) surely, the fixed valuation \ell=(1/2,1/2) yields \ell\cdot X=\ell\cdot Y=1/2, while the entropic projection score changes with the distribution of the projection. Positivity of q makes the head monotone in every outcome coordinate; the log-moment form makes it sensitive to dispersion and dependence.

## Key formulas

- $$q\cdot X=\sum_{j=1}^{d}q_jX_j,\qquad q\in\mathbb{R}^{d}_{+}.$$
- $$\operatorname{CE}_{\eta}(Z)=-\frac{1}{\eta}\log\mathbb{E}\left[\exp(-\eta Z)\right],\qquad \eta>0.$$
- $$C(X)=\int \operatorname{CE}_{\eta}(q\cdot X)\,\mu(dq,d\eta)=\mathbb{E}_{(q,\eta)\sim\mu}\left[-\frac{1}{\eta}\log\mathbb{E}\exp(-\eta q\cdot X)\right].$$
- $$X=\begin{cases}(1,0),&\text{with probability }1/2,\\(0,1),&\text{with probability }1/2,\end{cases}\qquad Y=(1/2,1/2)\ \text{surely},\qquad \ell=(1/2,1/2),\qquad \ell\cdot X=\ell\cdot Y=1/2.$$

## Implementation notes

(1) Integration point: use this as the final utility/value head in a multi-objective RL critic, a probabilistic world model, or a regression model whose output is a batch of K possible vector outcomes. The base network produces samples or particles X[b,k,:] with d objectives. Do not scalarize these objectives before the head.

(2) Pseudocode:
```
raw_q = parameter_or_network_output()              # [M,d]
q = softmax(raw_q, dim=-1)                         # q_m >= 0, sum_j q_mj = 1
eta = softplus(raw_eta) + eta_min                  # [M], eta_m > 0
z[b,m,k] = sum_j q[m,j] * X[b,k,j]                 # q_m dot X_bk
ce[b,m] = -(1/eta[m]) * logmeanexp(-eta[m]*z[b,m,:], dim=k)
score[b] = sum_m softmax(mixture_logits)[m] * ce[b,m]
```
Train by maximizing `score` for chosen actions in RL, or minimizing a task loss that uses `-score` for desirable stochastic predictions. For a supervised risk target, retain the vector target and compare the mixture score with a scalar preference label; alternatively use the score as a regularizer alongside per-coordinate losses. Stop gradients through neither q nor eta if they are intended to learn, but initialize q near uniform and eta near a small value so the model starts close to mean scalarization.

(3) The mathematical computation is the positive projection, log-mean-exp entropic certainty equivalent, and positive mixture. Estimate the inner expectation with K particles, trajectories, augmentations, or dropout samples. Estimate q and eta by gradient descent; constrain q with softmax and eta with softplus. Add a numerically stable implementation using `logsumexp` and optionally clip `eta * z` to avoid overflow.

(4) First experiment: train a small distributional actor-critic on a two-objective synthetic environment and compare a fixed weighted-sum critic, a distributional critic with mean aggregation, and this M=4 projection-mixture head at equal parameter count. Use the paper's diagnostic X-versus-Y construction plus environments with conflicting objectives and stochastic complementarities. Measure policy return on each coordinate, Pareto-front coverage, calibration under changed objective weights, and critic loss. Success is a higher worst-coordinate return or better Pareto coverage without reducing mean return, together with the head assigning different scores to the complementary lottery X and deterministic outcome Y even though their fixed linear value is identical.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.