△ Mechanism confirmed, baseline not beaten 2026

Hadamard-CLIP joint interaction head

Implementation & benchmark of arXiv:2608.17203 — Expressivity In Multimodal Contrastive Learning

Usefulness8/10

Difficulty4/10

Novelty7/10

Architecture Accuracy Sample efficiency Scalability Retrieval Embedding Transformer Attention Linear algebra Statistics Approximation theory Representation theory

Source paper: Expressivity In Multimodal Contrastive Learning arXiv:2608.17203 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the usual sum of pairwise modality similarities with a higher-order score based on the coordinatewise Hadamard product of all normalized modality embeddings. For modalities indexed by i=1,...,m, score a tuple using s(x_1,...,x_m)=\omega^ op(\bar g_1(x_1)\odot\cdots\odot\bar g_m(x_m)), where \omega is learned and \odot is coordinatewise multiplication. This adds explicit m-way interactions without concatenating raw features or introducing a joint encoder.

Formulas

$$s_{\mathrm{H}}(x_1,\ldots,x_m)=\omega^{\top}\bigl(\bar{g}_1(x_1)\odot\cdots\odot\bar{g}_m(x_m)\bigr),\qquad \exp\bigl(s_{\mathrm{H}}(x_1,\ldots,x_m)\bigr)$$

$$s_{\mathrm{P}}(x_1,\ldots,x_m)=\frac{1}{\tau}\sum_{1\le i<j\le m}\langle\bar g_i(x_i),\bar g_j(x_j)\rangle$$

$$\mathsf L_{\mathrm{joint}}=-\frac1N\sum_{n=1}^{N}\log\frac{\exp\left(s_{\mathrm H}(x_1^{(n)},\ldots,x_m^{(n)})/\tau\right)}{\sum_{k=1}^{N}\exp\left(s_{\mathrm H}(x_1^{(k)},x_2^{(n)},\ldots,x_m^{(n)})/\tau\right)}$$

$$\bar g_i(x_i)=\frac{h_i(x_i)}{\|h_i(x_i)\|_2+\epsilon},\qquad q(x_1,\ldots,x_m)=\frac{\exp\left(\omega^{\top}(\bar g_1(x_1)\odot\cdots\odot\bar g_m(x_m))/\tau\right)}{Z}$$

Mathematical statement

The paper's Hadamard-CLIP architecture uses normalized encoders \bar g_i:X_i o\mathbb R^d satisfying \|\bar g_i(x_i)\|_2=1, a learned vector \omega\in\mathbb R^d, and the coordinatewise product \odot. Its joint compatibility score is s_H(x_1,\ldots,x_m)=\omega^ op(\bar g_1(x_1)\odot\cdots\odot\bar g_m(x_m)), with exponential density or energy factor \exp(s_H). The pairwise-sum architecture instead uses s_P= au^{-1}\sum_{i<j}\langle\bar g_i(x_i),\bar g_j(x_j)\rangle. The paper proves that pairwise summation cannot represent arbitrary joint distributions for m\ge 3, while Hadamard-CLIP restores universal approximation of the joint under its encoder-capacity assumptions. The transferable property is that the product creates an interaction among all m modalities while each encoder remains independently precomputable. Here au>0 is a temperature, d is the embedding dimension, \omega is the learned interaction weight, and Z is the normalizing constant if the score is interpreted as an energy-based density.

Implementation notes

Integrate the head immediately after the modality-specific projection layers and before the contrastive logits. Let each encoder produce h_i with shape [batch,d], normalize rowwise to g_i=h_i/(||h_i||_2+epsilon), and maintain one learnable parameter omega with shape [d]. For every aligned batch, compute the joint interaction tensor p=g_1; for i=2,...,m update p=p*g_i using elementwise multiplication; then compute logits=(p*omega).sum(-1)/tau. Use the aligned tuple as the positive and construct negatives by independently permuting one modality, replacing one modality with every batch element, or forming a matrix of candidate tuples. Pseudocode is: g=[normalize(project_i(x_i)) for i in range(m)]; p=g[0]; for i in range(1,m): p=p*g[i]; joint_logits=(p*omega).sum(dim=-1)/tau; loss=CrossEntropy(joint_logits_matrix, arange(B)). Compute normalization and the Hadamard product exactly from the proposed architecture; no theoretical constants need estimation. Initialize omega to ones or a small random vector, parameterize tau as exp(log_tau), and monitor product underflow, gradient norms, and the variance of p. First test a synthetic three-modality binary or categorical dataset containing a parity/XOR variable: construct distributions with equal pairwise marginals but different three-way dependence, where pairwise scores should fail while the Hadamard model can fit the joint. Then test a small tri-modal encoder on aligned image-caption-label or image-audio-text data against pairwise-sum CLIP at equal embedding dimension, optimizer, batch size, and FLOPs. Success means lower held-out joint negative log-likelihood, higher retrieval accuracy when two modalities jointly retrieve the third, and improved performance on purely higher-order signals, with no material increase in embedding storage or encoding latency.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a reproducible Hadamard-CLIP toy MVP with exact parity-energy fitting, pairwise baseline, interaction-product scaling checks, and joint retrieval evaluation. All three mechanism predictions were confirmed: parity NLL gain matched the corrected closed form at rho=0.1–0.9 to numerical precision, product norms matched d^((1-m)/2) across m and d, and the parity coefficient matched atanh(rho) scaling. On rho=0.8 parity data, the pairwise model reached NLL 2.07944 and retrieval accuracy 0.50, while Hadamard reached NLL 1.71138 (true entropy) and accuracy 1.00; this demonstrates the promised higher-order expressivity on the synthetic signal.

Agent confidence: 9/10

Baseline: Pairwise energy: parity-sweep NLL fixed at log(8)=2.07944; rho=0.8 fitted NLL 2.0794413, retrieval accuracy 0.50
Idea: Hadamard energy: rho=0.8 fitted NLL 1.7113773, equal to true entropy, retrieval accuracy 1.00; analytical gain predictions matched observed gains with absolute error below 1e-8; product-norm sweep ratios were approximately 1.0

Limitations: Only a fixed synthetic three-binary-modality parity distribution was tested; modality encoders were hand-constructed rather than learned. No real multimodal retrieval dataset, wall-clock/FLOP comparison, finite-sample generalization study, GPU run, or comparison against a full CLIP training pipeline was performed.

How to run: python3 hadamard_clip_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.17203) Sep 2, 2026

Significantly WORSE than the tuned baseline (+41.9%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Binary interactions
Model: tri_modal_interaction_mlp
Paired seeds: 8
Baseline mean: 0.3442
Idea mean: 0.4885
Effect (Δ): 0.1444 (+41.9%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±13.2%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

The registered custom track is synthetic binary-interaction regression rather than a real image/audio/text dataset. No retrieval, wall-clock, FLOP, or embedding-storage comparison was run. The mechanism signature confirms only nonzero learned Hadamard product structure, not the flat-embedding norm scaling quantitatively.

How to run: python3 canonical_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

On the registered binary_interactions track, the end-to-end Hadamard interaction head was significantly worse than the matched pairwise-sum baseline. Baseline MSE was 0.344162 versus 0.488536 for Hadamard, with paired delta +0.144374 and permutation p=0.0081. The trained-model signature confirmed nonzero three-way product structure, but it did not produce a task-metric win.

Baseline: Pairwise-sum tri-modal interaction MLP, tuned best lr=0.03: mean test MSE 0.344162 across 8 seeds
Idea: Hadamard tri-modal interaction MLP, best lr=0.03: mean test MSE 0.488536 across 8 seeds
Paired delta mean: 0.1443737261
Permutation pvalue: 0.0081

Benchmark result