2-System Greedy Token Selection

Integrate this at the input of a ViT or multimodal transformer, immediately after patch or token embeddings are computed and before the expensive self-attention blocks. Given a sequence of N candidate embeddings e_i in R^d, select at most K tokens while enforcing a diversity constraint: a candidate is feasible only if it is at least w sequence positions from every selected token, or, in a semantic variant, has cosine similarity below a threshold with every selected token. Initialize S as the empty set and G as I_d. At each iteration, scan feasible candidates and compute the rank-one log-determinant marginal delta_i = log(1 + tau^{-1} e_i^T G^{-1} e_i). Select the candidate with largest delta_i, add it to S, and update G <- G + tau^{-1} e_i e_i^T. Stop at K tokens or when no candidate remains feasible. Use Cholesky rank-one updates rather than explicitly inverting G. The paper contributes the diminishing-returns greedy rule and the 1/3 guarantee under a 2-system; the implementation estimates utility from current token embeddings. For imperfect or noisy embeddings, average each candidate's marginal over two stochastic augmentations or maintain an exponential moving average of marginals. First test on CIFAR-100 with DeiT-Tiny or ViT-Tiny, retaining 25%, 50%, and 75% of patches. Compare against random pruning, learned top-k gates, and cosine-diversity greedy at equal selector FLOPs. Success means higher accuracy at a fixed retained-token count, equal accuracy with at least 1.5x lower attention FLOPs, and lower selection variance across augmentations.