Trace-Polytope Diversity Regularizer

Integrate the regularizer after the final maxout or monotone maxout layer, using branch coefficient vectors rather than only scalar activations. For a one-layer maxout unit, \(a_T\) is the branch weight vector. For a multilayer network, reconstruct an approximate trace by backtracking the winning predecessor index at every max gate and composing the selected affine coefficients through the network. During the forward pass, save each max operation's argmax index. For every minibatch input \(x_b\), backtrack its winning trace \(T_b\); let \(\mathcal{A}\) be the set of unique winning traces and let \(a_T\) denote their coefficient vectors. Deduplicate vectors whose Euclidean distance is below tolerance \(\epsilon\), then compute pairwise distances and add \(\lambda_{\mathrm{div}}\mathcal{L}_{\mathrm{div}}\), where \(\tau\) is a distance scale. Pseudocode: run the network and save argmax indices; for each sample backtrack the winner and compute \(a_T\); deduplicate active traces; calculate the exponential pairwise penalty; add it to task loss and backpropagate. The paper supplies the exact trace/polytope identity; the diversity penalty and minibatch active-trace estimator are the ML adaptation. Start with a small maxout MLP on CIFAR-10 or synthetic convex regression, comparing no regularizer and branch entropy regularization. Track validation loss, number of distinct active traces, pairwise coefficient distance, and inference FLOPs. Success means equal accuracy with fewer active traces, or improved accuracy at fixed branch count.