{
 "artifacts": null,
 "category": "regularization",
 "description": "Regularize an intermediate neural representation to preserve task-relevant information that is not already present in a skip connection, cache, memory state, or side feature. Instead of maximizing ordinary mutual information between input and representation, maximize the conditional information gain over the reference representation, preventing the network from spending capacity redundantly copying information that the reference path already supplies.",
 "formulas_latex": [
  "$$\\eta_{\\mathbb D}(\\mathcal N)=\\sup_{\\rho_A\\neq\\sigma_A}\\frac{\\mathbb D(\\mathcal N(\\rho_A)\\|\\mathcal N(\\sigma_A))}{\\mathbb D(\\rho_A\\|\\sigma_A)}$$",
  "$$\\eta_{\\mathbb D}^{\\mathrm{cond}}(K)=\\sup_{P_{XR},Q_{XR}}\\frac{\\mathbb D(P_{ZR}\\|Q_{ZR})-\\mathbb D(P_R\\|Q_R)}{\\mathbb D(P_{XR}\\|Q_{XR})-\\mathbb D(P_R\\|Q_R)},\\qquad Z\\sim K(\\cdot\\mid X)$$",
  "$$\\eta_{\\mathrm{MI}}(K)=\\sup_{P_{CX}}\\frac{I(C;Z)}{I(C;X)},\\qquad \\eta_{\\mathrm{MI}}^{\\mathrm{cond}}(K\\mid R)\\approx\\frac{I(C;Z\\mid R)}{I(C;X\\mid R)}$$",
  "$$\\mathcal L=\\mathcal L_{\\mathrm{task}}-\\lambda\\,\\widehat I(C;Z\\mid R)+\\beta\\,\\widehat I(C;R\\mid Z)$$"
 ],
 "id": 224,
 "implementation": "Integrate this at a residual block or Transformer layer. Let X be the incoming token representation, Z the output of the attention-plus-MLP block, R the detached residual stream X or an external memory/KV summary, and C the prediction target: the next token for language modeling or the class label for classification. Add a conditional-information preservation term using a variational classifier q_phi(c|z,r). A practical lower bound is \\(\\widehat I(C;Z|R)=\\mathbb E[\\log q_\\phi(C|Z,R)]-\\mathbb E[\\log q_\\phi(C|R)]\\); train q_phi jointly, while the second term is a reference-only baseline. Use the loss \\(L=L_task-\\lambda\\widehat I(C;Z|R)+\\beta\\widehat I(C|R|Z)\\), where the optional third term discourages Z from becoming merely a copy of R. Pseudocode: compute X; set R=stop_gradient(X) or the cached memory; compute Z=Block(X); obtain logits from q_phi([Z,R]) and q_phi(R); add \\(-\\lambda(CE_{ref}-CE_{joint})\\) to the task loss; update both model and probe. The mathematical quantity estimated empirically is conditional mutual information, not the exact worst-case supremum contraction coefficient. First test on a 6-layer small Transformer trained on WikiText-2, comparing ordinary training, unconditional mutual-information regularization, and this conditional version at equal FLOPs. Measure validation perplexity, probe accuracy from Z after removing R, gradient variance, and whether the conditional method improves perplexity without increasing representation norm or parameter count.",
 "math_summary": "The paper's basic contraction coefficient is \\(\\eta_{\\mathbb D}(\\mathcal N)=\\sup_{\\rho\\neq\\sigma}\\mathbb D(\\mathcal N(\\rho)\\|\\mathcal N(\\sigma))/\\mathbb D(\\rho\\|\\sigma)\\), where \\(\\mathcal N\\) is a channel and \\(\\mathbb D\\) is a divergence. The conditional version subtracts distinguishability already present in a reference system \\(R\\): for paired joint distributions \\(P_{XR},Q_{XR}\\) and a channel \\(K:X\\to Z\\), use \\(\\eta_{\\mathbb D}^{\\mathrm{cond}}(K)=\\sup [\\mathbb D(P_{ZR}\\|Q_{ZR})-\\mathbb D(P_R\\|Q_R)]/[\\mathbb D(P_{XR}\\|Q_{XR})-\\mathbb D(P_R\\|Q_R)]\\). For relative entropy, the paper identifies this conditional contraction with contraction of conditional mutual information. In the neural adaptation, \\(X\\) is the pre-block activation, \\(Z\\) is the post-block activation, \\(R\\) is a skip or memory representation, and \\(C\\) is a task label or future-token context; the target quantity is \\(I(C;Z\\mid R)\\), rather than ordinary \\(I(C;Z)\\).",
 "math_tags": [
  "information-theory",
  "probability"
 ],
 "ml_areas": [
  "transformer",
  "regularization",
  "training-dynamics"
 ],
 "paper": {
  "arxiv_id": "2608.27171",
  "arxiv_url": "https://arxiv.org/abs/2608.27171",
  "summary_what_math_gives_to_ml": "The paper introduces contraction coefficients that explicitly subtract distinguishability already available in a quantum reference system, and proves composition bounds for mutual-information and trace-distance contraction. The transferable asset is not the quantum formalism itself, but a principled way to measure how much information a layer preserves beyond a skip connection, side input, memory state, or previously computed representation. In neural networks this suggests conditional information-preservation losses and layerwise contraction budgets, with composition rules that distinguish information newly processed by a block from information already carried by the reference path.",
  "title": "Conditional contraction coefficients and their applications to quantum networks",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 6,
  "usefulness": 6
 },
 "solves": [
  "accuracy",
  "stability",
  "generalization"
 ],
 "title": "Reference-Subtracted Information Preservation",
 "url": "https://synthcore.org/idea/224/reference-subtracted-information-preservation",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}