Conditional-information-preserving pooling

Integrate this into a hierarchical Transformer token-merging layer or a graph-NN pooling layer. Partition tokens into fixed blocks of three, compute a pooled token $s_j=\sum_{k=1}^{3}\alpha_{jk}h_{jk}$ with learned softmax weights $\alpha_{jk}$, and initialize the weights to uniform averaging. For each minibatch, sample a remote block $R$ separated from source block $L$ by at least one block. Form $S=\operatorname{pool}(L)$ and train two small predictor heads: $q_\phi(r\mid L,S)$, which sees the microscopic source block and pooled token, and $q_\psi(r\mid S)$, which sees only the pooled token. Add $\lambda\max(0,\widehat I-\tau)$ to the task loss, with $\widehat I=B^{-1}\sum_i[\log q_\phi(r_i\mid l_i,s_i)-\log q_\psi(r_i\mid s_i)]$, information budget $\tau$, and minibatch size $B$. The predictors should be optimized to estimate the gap; use stop-gradient or alternating updates when necessary so the pooling layer cannot minimize the term merely by making both predictors poor. For image features, let $R$ be a distant patch and predict its class or projected representation; for language, let $R$ be a future token span and use a small autoregressive auxiliary head. The paper-derived object is the conditional-information sufficiency criterion; the neural predictors empirically estimate it. First test on CIFAR-10 with a four-stage hierarchical ViT, comparing average pooling, learned attention pooling, and information-preserving pooling at equal token counts and FLOPs. Measure validation accuracy, accuracy after 4x token reduction, and the held-out predictor gap. Success is higher accuracy at the same sequence length and a lower remote-prediction gap than ordinary pooling, particularly under local patch corruption.