Reference-Subtracted Information Preservation

Integrate this at a residual block or Transformer layer. Let X be the incoming token representation, Z the output of the attention-plus-MLP block, R the detached residual stream X or an external memory/KV summary, and C the prediction target: the next token for language modeling or the class label for classification. Add a conditional-information preservation term using a variational classifier q_phi(c|z,r). A practical lower bound is \(\widehat I(C;Z|R)=\mathbb E[\log q_\phi(C|Z,R)]-\mathbb E[\log q_\phi(C|R)]\); train q_phi jointly, while the second term is a reference-only baseline. Use the loss \(L=L_task-\lambda\widehat I(C;Z|R)+\beta\widehat I(C|R|Z)\), where the optional third term discourages Z from becoming merely a copy of R. Pseudocode: compute X; set R=stop_gradient(X) or the cached memory; compute Z=Block(X); obtain logits from q_phi([Z,R]) and q_phi(R); add \(-\lambda(CE_{ref}-CE_{joint})\) to the task loss; update both model and probe. The mathematical quantity estimated empirically is conditional mutual information, not the exact worst-case supremum contraction coefficient. First test on a 6-layer small Transformer trained on WikiText-2, comparing ordinary training, unconditional mutual-information regularization, and this conditional version at equal FLOPs. Measure validation perplexity, probe accuracy from Z after removing R, gradient variance, and whether the conditional method improves perplexity without increasing representation norm or parameter count.