Unverified 2026

Compositional Contraction Budget for Residual Blocks

Implementation & benchmark of arXiv:2608.27171 — Conditional contraction coefficients and their applications to quantum networks

Usefulness5/10

Difficulty6/10

Novelty7/10

Dynamics Stability Scalability Speedup Transformer Training dynamics Inference speedup Pruning Information theory Probability Dynamical systems

Source paper: Conditional contraction coefficients and their applications to quantum networks arXiv:2608.27171 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Estimate how strongly each neural block contracts distinguishability and use the paper's weighted composition inequality to allocate depth, residual strength, or precision where information is actually preserved. Blocks that strongly contract information beyond the reference path receive a smaller residual gate, higher numerical precision, or are replaced by a cheaper identity-like operation.

Formulas

$$\eta_{\mathrm{MI}}(\mathcal N_{A\to BE})\leq\eta_{\mathcal R}\,\eta_{\mathrm{MI}}(\mathcal M_{A\to BD})+(1-\eta_{\mathcal R})\,\eta_{\mathrm{MI}}(\mathcal M_{A\to B})$$

$$\eta_D(\mathcal N_{A\to BE})\leq\eta_{\mathcal R}\,\eta_D(\mathcal M_{A\to BD})+(1-\eta_{\mathcal R})\eta_D(\mathcal M_{A\to B}),\qquad \eta_{\mathcal R}=\eta_D^\Delta(\mathcal R_{D\to E})$$

$$\eta_{\mathrm{MI}}(\mathcal N^{\otimes n})\leq1-(1-\eta_D^\Delta(\mathcal N))^n\leq n\,\eta_D^\Delta(\mathcal N)$$

$$g_{\ell+1}=\operatorname{clip}\left(g_\ell\exp\left[-\alpha\left(\widehat b_\ell-b_0\right)\right],g_{\min},g_{\max}\right),\quad Z_\ell=R_\ell+g_\ellF_\ell(R_\ell)$$

Mathematical statement

Proposition 11 gives a composition bound for a network $\mathcal N$ formed from a main channel $\mathcal M$ and an auxiliary channel $\mathcal R$: $\eta_{\mathrm{MI}}(\mathcal N)\leq \eta_{\mathcal R}\eta_{\mathrm{MI}}(\mathcal M_{A\to BD})+(1-\eta_{\mathcal R})\eta_{\mathrm{MI}}(\mathcal M_{A\to B})$. Here $\eta_{\mathcal R}=\eta_D^\Delta(\mathcal R_{D\to E})$ is the conditional trace-distance contraction of the auxiliary channel, while the two mutual-information coefficients measure contraction when its output is retained or discarded. Corollary 13 gives $\eta_{\mathrm{MI}}(\mathcal N^{\otimes n})\leq1-(1-\eta_D^\Delta(\mathcal N))^n\leq n\eta_D^\Delta(\mathcal N)$. The adaptation treats a residual or memory branch as $\mathcal R$ and uses this bound as a measurable information-loss budget across depth.

Implementation notes

Apply this to residual Transformer blocks, recurrent state updates, or MoE expert branches. For block ℓ, define the main channel $\mathcal M$ as the nonlinear transformation $F_\ell$, and the reference channel $\mathcal R$ as the residual identity, skip summary, or memory update. Estimate a classical trace-distance contraction by sampling pairs of activation distributions: for minibatch subsets with labels or random binary partitions, compute $\widehat D_{TV}(P,Q)=\frac12\|h(P)-h(Q)\|_1$, where h is a small discriminator trained to distinguish the two distributions. Set $\widehat\eta_\ell=\widehat D_{TV}(Z_P,Z_Q)/(\widehat D_{TV}(X_P,X_Q)+\epsilon)$, and estimate separate values with and without the residual or memory feature exposed to h. Form the Proposition-11-style budget $\widehat b_\ell=\widehat\eta_R\widehat\eta_{MI}^{with}+(1-\widehat\eta_R)\widehat\eta_{MI}^{without}$. Update the residual gate using the displayed exponential rule; high contraction lowers the nonlinear gate, while low contraction permits more computation. The theorem supplies the composition form, whereas all contraction coefficients are empirical estimates. Start with a 6-layer CIFAR-10 ViT or 4-layer WikiText-2 Transformer and compare fixed residual gates, LayerScale, and this adaptive gate. Success is lower gradient variance and faster validation-loss descent at equal FLOPs, or pruning or cheapening high-contraction blocks with less than 1% accuracy or perplexity degradation.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.