Zonotope Volume Diversity Regularizer
Implementation & benchmark of arXiv:2608.12681 — Volume and Projection Inequalities I: Zonoids and Courtade's Conjecture
Source paper: Volume and Projection Inequalities I: Zonoids and Courtade's Conjecture arXiv:2608.12681 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Represent a collection of neural directions as generators of a zonotope and reward the volume spanned by their subsets. The objective favors complementary, non-collapsed vectors rather than merely pairwise-separated vectors, making it suitable for attention heads, MoE expert signatures, or embedding prototypes. Use normalized generators and positive gates so the regularizer cannot be increased trivially by scaling.
Formulas
Mathematical statement
For generators a_1,...,a_n in R^d, the zonotope is Z=sum_i [0,a_i]. Its d-dimensional volume is the determinant polynomial f(w)=sum_{S subseteq [n], |S|=d} |det(A_S)| product_{i in S} w_i, where A_S is the d by d matrix formed from selected generators and w_i are nonnegative gate weights. For an unweighted zonotope, |Z|=f(1)=sum_{|S|=d}|det(A_S)|. The paper studies the Rayleigh inequality partial_i f(w) partial_j f(w) >= f(w) partial_ij f(w), equivalent to partial_ij log f(w) <= 0 for i not equal to j; this is the differential form of log-submodularity. The paper proves the property when the degree or codegree is at most 3, but constructs counterexamples when both are at least 4. In the neural adaptation, a_i are normalized learned feature directions, w_i are positive module gates, and the regularizer is -log(f(w)+epsilon).
Implementation notes
(1) Integration point: attach the regularizer to n vectors that should diversify, such as attention-head output directions, MoE expert keys, or prototype embeddings. Let the raw vectors be h_i in R^D. Use a fixed random projection P in R^(r by D), with r=4 or 8, define a_i=P h_i divided by its Euclidean norm plus 1e-8, and use positive gates w_i=softplus(g_i)+1e-4. Add alpha times L_div to the task loss during training. (2) Pseudocode: H=module_vectors(); A=normalize(H@P.T); w=softplus(g)+floor; f=0; di=0; dij=0; for S in subsets_of_size_r: c=abs(det(A[S])); q=prod(w[S]); f+=c*q; for i in S: di[i]+=c*q/w[i]; for i<j in S: dij[i,j]+=c*q/(w[i]*w[j]); Ldiv=-log(f+eps); defect=sum(relu((f*dij-di*dj)/(f*f+eps))^2); loss=task_loss+alpha*Ldiv+lambda_R*defect. (3) Compute determinant sums exactly when n is small. For larger n, sample M subsets of size r and use importance-weighted estimates of f and its derivatives. The determinant formula supplies the objective; Rayleigh defects are only empirical because the paper gives high-dimensional counterexamples. (4) First experiment: train ViT-Tiny on CIFAR-100 with four or eight attention heads, comparing the baseline with the same model and regularizer at alpha in {1e-4,1e-3,1e-2}. Keep FLOPs and parameters fixed. Log effective rank of the projected head-direction Gram matrix, mean pairwise cosine similarity, estimated zonotope volume, validation accuracy, and seed variance. Success means less head collapse and improved or equal validation accuracy at matched compute; reject the method if volume increases only through gate growth or if accuracy is unchanged.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.