Contravariance Theory: Strong Alignment for Minimal Solutions to Hard Tasks
arXiv:2607.08561
2026
Architecture
2 ideas extracted · analyzed Aug 30, 2026
What the math gives to ML
The paper gives a constructive way to distinguish task-visible nonlinear units from removable or redundant units, using gate crossings, outgoing connectivity, and affine or algebraic nonredundancy. Its strongest transferable asset is a task-dependent width notion, m_l(epsilon), rather than pruning by magnitude or activation frequency alone. It also provides a mechanism for converting approximate affine representation matching into coordinate-level matching, suggesting a practical cross-model distillation or model-merging objective based on permutation and rescaling of hidden axes. The most promising experiments are task-aware ReLU pruning and alignment-constrained student training, with the paper's exact assumptions treated as empirical diagnostics rather than universal guarantees.
Ideas from this paper
Unverified
2026
Use a teacher and student with comparable hierarchical layers, first matching their representations through a learned affine map and then explicitly encouraging one-to-one coordinate correspondence up to permutation and positive rescaling. This operationalizes the paper's weak-to-strong alignment mechanism for representation distillation, model merging, or stabilizing independently trained replicas.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Prune hidden units only after testing whether their nonlinear gate is task-visible and downstream-used. For ReLU, a unit is removable when its preactivation does not cross zero on the task patch or its outgoing weight column is zero; this is a more structural criterion than weight magnitude and can be applied during width search or post-training compression.
Useful6/10
Difficulty3/10
Novelty5/10