# Nominal-Safe Scalar Association Regularizer

- ID: 1447
- Canonical URL: https://synthcore.org/idea/1447/nominal-safe-scalar-association-regularizer
- API JSON: https://synthcore.org/api/idea/1447.json
- API Markdown: https://synthcore.org/api/idea/1447.md
- Verification status: unverified
- Source: [arXiv:2607.26508](https://arxiv.org/abs/2607.26508)
- Category: regularization
- Solves: accuracy, generalization
- ML areas: regularization, embedding, moe-routing
- Math tags: statistics, information-theory, probability
- Ratings: usefulness 5/10; difficulty 3/10; novelty 6/10

## Idea description

Add an auxiliary objective that makes a selected scalar neural representation informative about a categorical variable while remaining invariant to permutations of the category labels. Estimate class posteriors from the scalar through a small softmax probe, and reward conditional posterior concentration above the marginal class-concentration baseline. The regularizer can be applied to bottleneck coordinates, uncertainty scores, diffusion time embeddings, or scalar MoE routing statistics.

## Mathematical statement

For a real-valued scalar X and categorical variable Y with k categories, the paper defines p_j=P(Y=j), the marginal probability of category j, and g_j(x)=P(Y=j\mid\sigma(X)), the conditional posterior given X, with \sum_{j=1}^k g_j(x)=1. It defines h(x)=\sum_{j=1}^k g_j(x)^2 and A=E[h(X)]=E[\sum_j P(Y=j\mid X)^2], while B=\sum_{j=1}^k p_j^2 is the baseline posterior concentration when X carries no information. The association numerator is A-B: it is zero under independence and positive when X changes the class posterior. In the neural adaptation, use a differentiable probe q_j(x)=\operatorname{softmax}(a_j x+b_j) and minibatch estimates \hat p_j=\frac{1}{n}\sum_i y_{ij}, \hat A=\frac{1}{n}\sum_i\sum_j q_j(x_i)^2. Normalize by the available headroom 1-\sum_j\hat p_j^2, giving R=(\hat A-\sum_j\hat p_j^2)/(1-\sum_j\hat p_j^2+\varepsilon). Here y_{ij}\in\{0,1\} is the one-hot target, n is the minibatch size, and \varepsilon>0 prevents division by zero. Unlike integer-coded correlation, permuting category indices only permutes the summands and leaves R unchanged.

## Key formulas

- $$p_j:=P(Y=j),\qquad g_j(X)=P\!\left(Y=j\mid\sigma(X)\right),\qquad \sum_{j=1}^k g_j(x)=1,$$
- $$h(x):=\sum_{j=1}^k g_j(x)^2,\qquad A:=E[h(X)]=E\!\left[\sum_{j=1}^k P(Y=j\mid X)^2\right],\qquad B:=\sum_{j=1}^k p_j^2,$$
- $$\widehat R(x_{1:n},y_{1:n})=\frac{\widehat A-\widehat B}{1-\widehat B+\varepsilon},\qquad \widehat A=\frac1n\sum_{i=1}^n\sum_{j=1}^k q_j(x_i)^2,\quad \widehat B=\sum_{j=1}^k\widehat p_j^2,\quad \widehat p_j=\frac1n\sum_{i=1}^n y_{ij},$$
- $$q_j(x)=\frac{\exp(a_jx+b_j)}{\sum_{\ell=1}^k\exp(a_\ell x+b_\ell)},\qquad \mathcal L=\mathcal L_{\mathrm{task}}-\lambda\,\widehat R.$$

## Implementation notes

Integrate this at a scalar representation tensor, not at the main classifier output. Choose a scalar x_i from each example, such as one bottleneck coordinate, a learned uncertainty score, the mean attention logit for a feature group, or the pre-softmax score used by an MoE router. Attach a lightweight probe with parameters a[1..k] and b[1..k], where q_i=softmax(a*x_i+b). For each minibatch, obtain one-hot categorical targets y_i, compute p_hat=mean(y_i, dim=0), B_hat=sum(p_hat**2), A_hat=mean(sum(q_i**2, dim=1)), and R_hat=(A_hat-B_hat)/(1-B_hat+epsilon). Optimize L_task - lambda*R_hat; detach p_hat if batch-statistic gradients create instability, or use an exponential moving average p_ema for B_hat. The probe should normally be optimized jointly, but run an ablation with probe parameters stopped after a warmup to distinguish representation learning from probe memorization. The paper supplies the posterior-concentration construction; the implementation estimates expectations empirically and uses a stable softmax. Start with a small MLP on CIFAR-10 or a UCI tabular dataset, using one scalar bottleneck and a standard cross-entropy baseline. Compare random category permutations, multiple lambda values, and a control using integer-encoded Pearson correlation. Success means unchanged performance under label permutation, higher test accuracy or calibration at equal parameter count, and a measurable increase in held-out association of the scalar without simply increasing the probe's training accuracy.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.