Expressivity In Multimodal Contrastive Learning
arXiv:2608.17203
2026
Architecture
1 ideas extracted · analyzed Sep 1, 2026
What the math gives to ML
The paper identifies a concrete expressivity gap in multimodal contrastive architectures: summing pairwise similarities cannot represent arbitrary higher-order dependencies when there are three or more modalities, even though it can match every pairwise conditional. Its proposed Hadamard-CLIP closes this gap with only one learned vector applied to the coordinatewise product of independently encoded modalities, retaining precomputable per-modality embeddings and dot-product-style retrieval. This is directly transferable as a drop-in replacement for the joint score in multiway CLIP or multimodal retrieval training. The highest-value experiment is to compare pairwise-sum and Hadamard scores on synthetic distributions with identical pairwise marginals but different genuine three-way interactions, followed by a real multimodal retrieval benchmark.
Ideas from this paper
△ Mechanism confirmed, baseline not beaten
2026
Replace the usual sum of pairwise modality similarities with a higher-order score based on the coordinatewise Hadamard product of all normalized modality embeddings. For modalities indexed by i=1,...,m, score a tuple using s(x_1,...,x_m)=\omega^\top(\bar g_1(x_1)\odot\cdots\odot\bar g_m(x_m)), where \omega is learned and \odot is coordinatewise multiplication. This adds explicit m-way interactions without concatenating raw features or introducing a joint encoder.
Useful8/10
Difficulty4/10
Novelty7/10