# Conditional-information-preserving pooling

- ID: 3040
- Canonical URL: https://synthcore.org/idea/3040/conditional-information-preserving-pooling
- API JSON: https://synthcore.org/api/idea/3040.json
- API Markdown: https://synthcore.org/api/idea/3040.md
- Verification status: mechanism_failed
- Source: [arXiv:2609.02141](https://arxiv.org/abs/2609.02141)
- Category: architecture
- Solves: accuracy, memory, generalization
- ML areas: transformer, graph-nn, loss, regularization
- Math tags: information-theory, probability, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace ordinary token merging or graph pooling with a learned block map whose output preserves information about a remote target conditioned on the surrounding coarse representation. The paper's majority-spin counterexample gives a concrete failure mode: two microscopic configurations mapped to the same pooled token can imply different predictions for distant variables.

## Mathematical statement

The paper defines a local majority coarse-graining channel on a three-spin block by $\left(\mathcal{E}_{j}\right)^{s^{\prime}}_{a,b,c}=\delta_{s^{\prime},\operatorname{maj}(a,b,c)}$, where $a,b,c\in\{-1,+1\}$ are microscopic spins, $s'\in\{-1,+1\}$ is the retained coarse spin, $\delta$ is the Kronecker delta, and $\operatorname{maj}$ is the sign of the sum. The key diagnostic is the conditional-mutual-information equality $I(L:R\mid s=+1)=I(\operatorname{maj}(L):R\mid s=+1)$, where $L$ and $R$ are microscopic left and right triples and $s$ is a conditioning logical spin. Equality means the pooled statistic is sufficient for predicting $R$ given $s$; failure means pooling discarded remote-relevant information. The explicit failure is $P(s_L=+1\mid L=(+1,+1,+1))=\frac{(1-p)^3}{(1-p)^3+p^3}$ versus $P(s_L=+1\mid L=(+1,+1,-1))=1-p$ for $0<p<1/2$, although both inputs have majority $+1$. For neural features, estimate the information gap with $\widehat I(L:R\mid S)=B^{-1}\sum_i[\log q_\phi(r_i\mid l_i,s_i)-\log q_\psi(r_i\mid s_i)]$, where $S$ is the learned pooled token, $q_\phi$ sees microscopic features and $S$, $q_\psi$ sees only $S$, and $B$ is minibatch size.

## Key formulas

- $$\left(\mathcal{E}_{j}\right)^{s^{\prime}}_{a,b,c}=\delta_{s^{\prime},\operatorname{maj}(a,b,c)}$$
- $$I(L:R|s=+1)=I(\operatorname{maj}(L):R|s=+1)$$
- $$P(s_L=+1|L=(+1,+1,+1))=\frac{(1-p)^3}{(1-p)^3+p^3},\qquad P(s_L=+1|L=(+1,+1,-1))=1-p$$
- $$\widehat I(L:R\mid S)=\frac{1}{B}\sum_{i=1}^{B}\left[\log q_{\phi}(r_i\mid l_i,s_i)-\log q_{\psi}(r_i\mid s_i)\right]$$

## Implementation notes

Integrate this into a hierarchical Transformer token-merging layer or a graph-NN pooling layer. Partition tokens into fixed blocks of three, compute a pooled token $s_j=\sum_{k=1}^{3}\alpha_{jk}h_{jk}$ with learned softmax weights $\alpha_{jk}$, and initialize the weights to uniform averaging. For each minibatch, sample a remote block $R$ separated from source block $L$ by at least one block. Form $S=\operatorname{pool}(L)$ and train two small predictor heads: $q_\phi(r\mid L,S)$, which sees the microscopic source block and pooled token, and $q_\psi(r\mid S)$, which sees only the pooled token. Add $\lambda\max(0,\widehat I-\tau)$ to the task loss, with $\widehat I=B^{-1}\sum_i[\log q_\phi(r_i\mid l_i,s_i)-\log q_\psi(r_i\mid s_i)]$, information budget $\tau$, and minibatch size $B$. The predictors should be optimized to estimate the gap; use stop-gradient or alternating updates when necessary so the pooling layer cannot minimize the term merely by making both predictors poor. For image features, let $R$ be a distant patch and predict its class or projected representation; for language, let $R$ be a future token span and use a small autoregressive auxiliary head. The paper-derived object is the conditional-information sufficiency criterion; the neural predictors empirically estimate it. First test on CIFAR-10 with a four-stage hierarchical ViT, comparing average pooling, learned attention pooling, and information-preserving pooling at equal token counts and FLOPs. Measure validation accuracy, accuracy after 4x token reduction, and the held-out predictor gap. Success is higher accuracy at the same sequence length and a lower remote-prediction gap than ordinary pooling, particularly under local patch corruption.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built and ran a reproducible majority-spin conditional-information toy experiment plus a small learned scalar-pooling remote-prediction comparison. The exact check found I(L:R)=0.3043 bits versus I(majority(L):R)=0.2933 bits, with 0.0110 bits discarded and clearly different posteriors for two same-majority inputs. However, learned pooling did not beat mean pooling on held-out remote NLL or accuracy, so the proposed neural method has no demonstrated win here.

### Mechanism check

- Verdict: Built and ran a reproducible majority-spin conditional-information toy experiment plus a small learned scalar-pooling remote-prediction comparison. The exact check found I(L:R)=0.3043 bits versus I(majority(L):R)=0.2933 bits, with 0.0110 bits discarded and clearly different posteriors for two same-majority inputs. However, learned pooling did not beat mean pooling on held-out remote NLL or accuracy, so the proposed neural method has no demonstrated win here.
- Confidence: 9/10
- Limitations: Only a synthetic noisy-spin system was tested; no CIFAR-10, hierarchical Transformer, local corruption, task-loss training, explicit information-budget regularizer, multiple random seeds, or statistically repeated trials were run. The neural demo is a small auxiliary-prediction test rather than an end-to-end pooling benchmark.

## Artifacts

- [report.md](https://synthcore.org/code/1222/report.md)
- [toy_experiment.py](https://synthcore.org/code/1222/toy_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/1222)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.