# Reference-Subtracted Information Preservation

- ID: 224
- Canonical URL: https://synthcore.org/idea/224/reference-subtracted-information-preservation
- API JSON: https://synthcore.org/api/idea/224.json
- API Markdown: https://synthcore.org/api/idea/224.md
- Verification status: unverified
- Source: [arXiv:2608.27171](https://arxiv.org/abs/2608.27171)
- Category: regularization
- Solves: accuracy, stability, generalization
- ML areas: transformer, regularization, training-dynamics
- Math tags: information-theory, probability
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Regularize an intermediate neural representation to preserve task-relevant information that is not already present in a skip connection, cache, memory state, or side feature. Instead of maximizing ordinary mutual information between input and representation, maximize the conditional information gain over the reference representation, preventing the network from spending capacity redundantly copying information that the reference path already supplies.

## Mathematical statement

The paper's basic contraction coefficient is \(\eta_{\mathbb D}(\mathcal N)=\sup_{\rho\neq\sigma}\mathbb D(\mathcal N(\rho)\|\mathcal N(\sigma))/\mathbb D(\rho\|\sigma)\), where \(\mathcal N\) is a channel and \(\mathbb D\) is a divergence. The conditional version subtracts distinguishability already present in a reference system \(R\): for paired joint distributions \(P_{XR},Q_{XR}\) and a channel \(K:X\to Z\), use \(\eta_{\mathbb D}^{\mathrm{cond}}(K)=\sup [\mathbb D(P_{ZR}\|Q_{ZR})-\mathbb D(P_R\|Q_R)]/[\mathbb D(P_{XR}\|Q_{XR})-\mathbb D(P_R\|Q_R)]\). For relative entropy, the paper identifies this conditional contraction with contraction of conditional mutual information. In the neural adaptation, \(X\) is the pre-block activation, \(Z\) is the post-block activation, \(R\) is a skip or memory representation, and \(C\) is a task label or future-token context; the target quantity is \(I(C;Z\mid R)\), rather than ordinary \(I(C;Z)\).

## Key formulas

- $$\eta_{\mathbb D}(\mathcal N)=\sup_{\rho_A\neq\sigma_A}\frac{\mathbb D(\mathcal N(\rho_A)\|\mathcal N(\sigma_A))}{\mathbb D(\rho_A\|\sigma_A)}$$
- $$\eta_{\mathbb D}^{\mathrm{cond}}(K)=\sup_{P_{XR},Q_{XR}}\frac{\mathbb D(P_{ZR}\|Q_{ZR})-\mathbb D(P_R\|Q_R)}{\mathbb D(P_{XR}\|Q_{XR})-\mathbb D(P_R\|Q_R)},\qquad Z\sim K(\cdot\mid X)$$
- $$\eta_{\mathrm{MI}}(K)=\sup_{P_{CX}}\frac{I(C;Z)}{I(C;X)},\qquad \eta_{\mathrm{MI}}^{\mathrm{cond}}(K\mid R)\approx\frac{I(C;Z\mid R)}{I(C;X\mid R)}$$
- $$\mathcal L=\mathcal L_{\mathrm{task}}-\lambda\,\widehat I(C;Z\mid R)+\beta\,\widehat I(C;R\mid Z)$$

## Implementation notes

Integrate this at a residual block or Transformer layer. Let X be the incoming token representation, Z the output of the attention-plus-MLP block, R the detached residual stream X or an external memory/KV summary, and C the prediction target: the next token for language modeling or the class label for classification. Add a conditional-information preservation term using a variational classifier q_phi(c|z,r). A practical lower bound is \(\widehat I(C;Z|R)=\mathbb E[\log q_\phi(C|Z,R)]-\mathbb E[\log q_\phi(C|R)]\); train q_phi jointly, while the second term is a reference-only baseline. Use the loss \(L=L_task-\lambda\widehat I(C;Z|R)+\beta\widehat I(C|R|Z)\), where the optional third term discourages Z from becoming merely a copy of R. Pseudocode: compute X; set R=stop_gradient(X) or the cached memory; compute Z=Block(X); obtain logits from q_phi([Z,R]) and q_phi(R); add \(-\lambda(CE_{ref}-CE_{joint})\) to the task loss; update both model and probe. The mathematical quantity estimated empirically is conditional mutual information, not the exact worst-case supremum contraction coefficient. First test on a 6-layer small Transformer trained on WikiText-2, comparing ordinary training, unconditional mutual-information regularization, and this conditional version at equal FLOPs. Measure validation perplexity, probe accuracy from Z after removing R, gradient variance, and whether the conditional method improves perplexity without increasing representation norm or parameter count.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.