# Trace-Polytope Diversity Regularizer

- ID: 755
- Canonical URL: https://synthcore.org/idea/755/trace-polytope-diversity-regularizer
- API JSON: https://synthcore.org/api/idea/755.json
- API Markdown: https://synthcore.org/api/idea/755.md
- Verification status: unverified
- Source: [arXiv:2607.11540](https://arxiv.org/abs/2607.11540)
- Category: regularization
- Solves: accuracy, generalization, scalability
- ML areas: mlp, regularization, moe-routing
- Math tags: convex-analysis, geometry, combinatorics
- Ratings: usefulness 6/10; difficulty 5/10; novelty 8/10

## Idea description

Use the trace representation of a maxout network to regularize the geometry of its generated coefficient vectors. Encourage active traces to be diverse and nonredundant, so the model spends parameters on genuinely different supporting hyperplanes rather than branches that collapse to the same linear function.

## Mathematical statement

An STC has input gates corresponding to \(0,e_1,\ldots,e_d\), plus gates for vector addition, max gates for convex-hull union, and positive scalar gates for dilation. Let \(\mathcal{T}\) be the set of traces obtained by selecting one predecessor at every max gate, and let \(a_T\in\mathbb{R}^d\) be the coefficient vector of trace \(T\). Lemma 3.4 states \(f_\Phi(x)=\max_{T\in\mathcal{T}}a_T^\top x\) and \(P_\Phi=\operatorname{conv}\{a_T:T\in\mathcal{T}\}\). Thus the function depends on the convex hull of trace vectors, not on the raw number of branches. For a minibatch \(X=\{x_b\}_{b=1}^B\), define the active trace \(T_b=\arg\max_Ta_T^\top x_b\), deduplicate active vectors, and penalize near-duplicate traces. The regularizer targets redundant geometric capacity while leaving the maxout function class unchanged.

## Key formulas

- $$f_{\Phi}(x)=\max_{T\in\mathcal{T}}a_T^{\top}x,\qquad P_{\Phi}=\operatorname{conv}\{a_T:T\in\mathcal{T}\},$$
- $$T_b=\arg\max_{T\in\mathcal{T}}a_T^\top x_b,$$
- $$\mathcal{L}_{\mathrm{div}}=\frac{1}{|\mathcal{A}|(|\mathcal{A}|-1)}\sum_{T\neq T'\in\mathcal{A}}\exp\left(-\frac{\|a_T-a_{T'}\|_2^2}{\tau^2}\right),$$
- $$\mathcal{L}=\mathcal{L}_{\mathrm{task}}+\lambda_{\mathrm{div}}\mathcal{L}_{\mathrm{div}},$$

## Implementation notes

Integrate the regularizer after the final maxout or monotone maxout layer, using branch coefficient vectors rather than only scalar activations. For a one-layer maxout unit, \(a_T\) is the branch weight vector. For a multilayer network, reconstruct an approximate trace by backtracking the winning predecessor index at every max gate and composing the selected affine coefficients through the network. During the forward pass, save each max operation's argmax index. For every minibatch input \(x_b\), backtrack its winning trace \(T_b\); let \(\mathcal{A}\) be the set of unique winning traces and let \(a_T\) denote their coefficient vectors. Deduplicate vectors whose Euclidean distance is below tolerance \(\epsilon\), then compute pairwise distances and add \(\lambda_{\mathrm{div}}\mathcal{L}_{\mathrm{div}}\), where \(\tau\) is a distance scale. Pseudocode: run the network and save argmax indices; for each sample backtrack the winner and compute \(a_T\); deduplicate active traces; calculate the exponential pairwise penalty; add it to task loss and backpropagate. The paper supplies the exact trace/polytope identity; the diversity penalty and minibatch active-trace estimator are the ML adaptation. Start with a small maxout MLP on CIFAR-10 or synthetic convex regression, comparing no regularizer and branch entropy regularization. Track validation loss, number of distinct active traces, pairwise coefficient distance, and inference FLOPs. Success means equal accuracy with fewer active traces, or improved accuracy at fixed branch count.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.