✗ Failed on benchmark 2026

Canonical orbit search for symmetric pruning masks

Implementation & benchmark of arXiv:2607.05967 — Symmetric lexicographic symmetric-subset reverse search for the enumeration of circuits, cocircuits, and triangulations up to symmetry

Usefulness6/10
Difficulty5/10
Novelty8/10

Source paper: Symmetric lexicographic symmetric-subset reverse search for the enumeration of circuits, cocircuits, and triangulations up to symmetry arXiv:2607.05967 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Enumerate structured pruning masks only up to exact permutations of exchangeable channels, hidden units, or experts. Replace exhaustive mask search with canonical augmentation: retain a subset only when it is lexicographically smallest among all masks obtained by the model's symmetry group, while recursively generating only canonical predecessors.

Formulas

$$S<_{\mathrm{lex}}R\iff\min(S\mathbin{\triangle}R)\in S,$$
$$C(S):=\operatorname{lex\text{-}min}\{g(S):g\in\mathfrak G\},\qquad S^{-}:=S\setminus\{\max S\},$$
$$C(S)=S\ \Longrightarrow\ C(S^{-})=S^{-},$$
$$\operatorname{keep}(S)=\mathbf 1\!\left[S=\min_{g\in\mathfrak G}^{\mathrm{lex}}g(S)\right],\qquad \mathcal M_k/\mathfrak G=\{S\subseteq[n]:|S|=k,\ \operatorname{keep}(S)=1\}.$$

Mathematical statement

The paper defines lexicographic order for equal-cardinality subsets S,R contained in [n] by S <_{lex} R if and only if min(S triangle R) belongs to S, where S triangle R is the symmetric difference and min(emptyset) is infinity. A finite permutation group G contained in the symmetric group on n elements acts on subsets by g(S) = {g(i): i in S}; the canonical representative is C(S) = lex-min {g(S): g in G}. Lemma 2 states that if S = C(S) and S is nonempty, then S-minus = S minus {max S} also satisfies S-minus = C(S-minus). Here [n] indexes candidate channels or experts and k = |S| is the number retained. This predecessor-closure property means depth-first backtracking that expands only canonical subsets still reaches every orbit exactly once. For an ML mask, G must be an exact network symmetry: permuting equivalent hidden channels requires applying the inverse permutation to adjacent weight dimensions, and permuting identical MoE experts requires permuting both expert parameters and router-output columns.

Implementation notes

Integrate this into a discrete structured-pruning or architecture-search evaluator before expensive fine-tuning. Let indices 1 through n denote exchangeable channels in one hidden layer, or exchangeable experts in one MoE layer. First construct the exact symmetry group: for a simple MVP, use all permutations of channels within one homogeneous layer; for several groups of equivalent components, use the product of their permutation groups. Represent a candidate retained set S as a sorted integer tuple. Implement is_canonical(S) by applying every g in G, sorting the resulting tuple, and comparing it with S using the first differing element; retain S only if it is lexicographically smallest. Enumerate recursively from the empty set by appending an index larger than the current maximum, and call is_canonical on every child before recursing. Pseudocode: dfs(S): evaluate(S) if len(S)==k; for i in range(max(S)+1,n): T=sort(S+[i]); if is_canonical(T): dfs(T). The predecessor formula guarantees that canonical children have canonical parents, so this does not discard an orbit's only representative. Estimate the group size and orbit count empirically; the canonicality test is exact for the supplied permutations. For larger groups, use a stabilizer-chain or graph-canonical-labeling implementation, validated against exhaustive enumeration for small n. The first experiment should use a width-12 MLP on MNIST or CIFAR-10, enumerate all masks retaining k=6 channels, and compare exhaustive mask evaluation against canonical search using channel permutations. Measure evaluated masks, wall-clock search time, and best validation accuracy after identical one-shot-supernet scoring. Success is a reduction near the number of symmetry orbits with unchanged best accuracy; for a fully exchangeable layer, the 924 masks can collapse to one representative.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built an exact permutation-group canonicalizer, canonical DFS, orbit coverage checks, and a two-block symmetry experiment. The math checks passed: canonical representatives covered every mask orbit, DFS matched the canonical set, predecessor closure held, and the lexicographic order check passed. Under full exchangeability, 70 masks collapsed to 1; under two exchangeable blocks, 70 masks collapsed to 5 with identical best score and 14x fewer evaluated candidates. The reduction is real, but naive factorial canonicality testing was slower in wall-clock time than exhaustive scanning at this tiny size.

Agent confidence
9/10
Baseline
70 masks; two-block best score 12; exhaustive scan 0.000064 s
Idea
5 canonical representatives; best score 12; 14x evaluation reduction; canonical DFS 0.0264 s
Toy verification
n=7,k=3: 35 masks, 1 orbit, DFS/reps equal, orbit coverage true, predecessor closure true

Limitations: No neural-network training or real pruning accuracy experiment was run. The implementation enumerates explicit permutation groups, so factorial group construction and canonicality scans do not scale; a stabilizer-chain or graph-canonical-labeling backend is needed for larger channel counts.

How to run: python3 canonical_orbit.py && /home/maxwelhelp/main/bin/python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.05967)

Significantly WORSE than the tuned baseline (+12.0%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark
Tabular regression (Friedman)
Model
masked_mlp_tiny
Paired seeds
8
Baseline mean
185.9184
Idea mean
208.2031
Effect (Δ)
22.2847 (+12.0%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect
±5.7%
Mechanism
Not confirmed ✗
Practical effect
Hurts
Baseline tuning
swept over 3 configs
Limitations:

Only the tabular track was tested; no CNN, MoE, multi-block symmetry, stabilizer-chain backend, or exact weight-permutation-equivalent initialization/training protocol was tested. The experiment demonstrates that naive index-level channel exchangeability is not an exact symmetry of independently trained networks.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark latest Failed ✗

Agent confidence: 10/10

Built a local exact-orbit mask-search benchmark using the bench tabular track and a shared six-channel MLP, with a three-value learning-rate sweep on both baseline and idea sides and eight paired seeds. Canonical search reduced evaluated candidates from 20 to 1, but the idea's test MSE was 208.203 versus 185.918 for exhaustive baseline (paired delta +22.285, permutation p=0.0081), and trained-model mask spreads were nonzero for all eight seeds. The structural speedup is real, but exact channel symmetry was not present in independently trained hidden units, so the promised neural-training benefit did not transfer.

Baseline
Best lr=0.01; 20 masks; full 8-seed MSE mean 185.9184, std 13.3165
Evaluation reduction
20x fewer mask evaluations
Idea
Best lr=0.01; 1 canonical mask; full 8-seed MSE mean 208.2031, std 13.8050
Mechanism signature
Predicted one orbit and 20x reduction; observed reduction 20x, but trained mask metric range was 36.05-73.77 across seeds and nonzero for 8/8, confirmed=false
Paired delta
idea - baseline = +22.2847 MSE; idea wins 0/8; permutation p=0.0081

Benchmark result

Significantly WORSE than the tuned baseline (+12.0%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark
Tabular regression (Friedman)
Model
masked_mlp_tiny
Paired seeds
8
Baseline mean
185.9184
Idea mean
208.2031
Effect (Δ)
22.2847 (+12.0%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect
±5.7%
Mechanism
Not confirmed ✗
Practical effect
Hurts
Baseline tuning
swept over 3 configs
Limitations:

Only the tabular track was tested; no CNN, MoE, multi-block symmetry, stabilizer-chain backend, or exact weight-permutation-equivalent initialization/training protocol was tested. The experiment demonstrates that naive index-level channel exchangeability is not an exact symmetry of independently trained networks.

How to run: python3 bench_experiment.py

Artifacts

Implementation overview ⬇ Download all as ZIP 6 files · code, reports and structured results