✗ Failed on benchmark 2026

Task-Visible Axis Pruning

Implementation & benchmark of arXiv:2607.08561 — Contravariance Theory: Strong Alignment for Minimal Solutions to Hard Tasks

Usefulness6/10

Difficulty3/10

Novelty5/10

Architecture Memory Speedup Accuracy Pruning Mlp Cnn Inference speedup Approximation theory Geometry Linear algebra

Source paper: Contravariance Theory: Strong Alignment for Minimal Solutions to Hard Tasks arXiv:2607.08561 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Prune hidden units only after testing whether their nonlinear gate is task-visible and downstream-used. For ReLU, a unit is removable when its preactivation does not cross zero on the task patch or its outgoing weight column is zero; this is a more structural criterion than weight magnitude and can be applied during width search or post-training compression.

Formulas

$$m_{\ell}(\varepsilon)=\min\{k:\exists f\in\mathcal{F}_{\ell,\leq k}\text{ with }L(f)\leq\varepsilon\}.$$

$$z^{N}_{\ell}(x)=W^{N}_{\ell}h^{N}_{\ell-1}(x)+b^{N}_{\ell},\qquad h^{N}_{\ell}(x)=\rho(z^{N}_{\ell}(x)),\qquad \rho(t)=\max\{t,0\}.$$

$$\operatorname{Used}_{\ell,j}=\mathbf{1}\!\left[\min_{x\in C}z_{\ell,j}(x)\leq 0\leq\max_{x\in C}z_{\ell,j}(x)\right]\mathbf{1}\!\left[\|W_{\ell+1}[:,j]\|_2>0\right].$$

Mathematical statement

The paper defines the required used-axis budget as m_l(epsilon) = min{k: exists f in F_{l,<=k} with L(f) <= epsilon}, where L is task loss, epsilon is the allowed loss, F_{l,<=k} is the fixed macroarchitecture restricted to at most k used axes at layer l, and m_l(epsilon) is the smallest task-compatible used width. A ReLU axis j is used when its preactivation z_{l,j}(x) crosses the zero trace set on the task patch C and its outgoing column W_{l+1}[:,j] is nonzero. If z_{l,j} remains strictly positive or strictly negative on C, then rho(z_{l,j}) = max(z_{l,j},0) is affine or zero on C and can be absorbed into adjacent affine layers. The implementation uses empirical approximations of the zero-crossing condition and estimates m_l(epsilon) by pruning until validation loss exceeds epsilon.

Implementation notes

Integrate this into structured width pruning for ReLU MLPs, CNN channels, or Transformer MLP neurons. At every chosen hidden layer, collect a task patch C from a calibration set and record each unit's preactivation z_{l,j}(x) and outgoing column W_{l+1}[:,j]. Compute a robust crossing score using low and high quantiles rather than raw extrema: q^-_j = Q_{0.01}(z_{l,j}) and q^+_j = Q_{0.99}(z_{l,j}). Mark the unit task-visible if q^-_j <= delta and q^+_j >= -delta, with a small tolerance delta, and downstream-used if ||W[:,j]||_2 > eta. In pseudocode: (1) run calibration batches; (2) accumulate q^-_j, q^+_j, and outgoing norms; (3) form the mask u_j = 1[q^-_j <= 0 <= q^+_j]1[||W[:,j]|| > eta]; (4) remove units with u_j = 0; (5) fine-tune; (6) repeat until validation loss exceeds the target epsilon. For convolutional channels, apply the same test to the channel preactivation over batch, spatial positions, and calibration examples. The formula for m_l(epsilon) is used as a stopping rule, while quantiles, delta, and eta are empirical estimates. The first experiment should use a 4-layer ReLU MLP on CIFAR-10 or a small ViT MLP block, comparing magnitude pruning, activation-mean pruning, and this criterion at equal retained width. Success means lower parameter or FLOP count at the same validation accuracy, and a sharper identifiable loss jump when pruning below the estimated used-axis budget.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built a reproducible ReLU MLP pruning MVP with magnitude, activation-mean, and task-visible-axis criteria. The core structural claim was numerically verified: explicitly positive/negative axes were correctly classified by quantiles, and absorbing them into adjacent affine computation produced maximum error 1.8e-15. On digits, task-visible pruning gave a small accuracy advantage after fine-tuning at 75% and 25% retained width, but not at 50%, so the practical win is promising but not consistent.

Agent confidence: 7/10

Baseline: Unpruned accuracy 0.9711; validation loss 0.1298.
Idea: Task-visible accuracy after 3 fine-tuning epochs: 0.9756 at 75% width, 0.9600 at 50%, and 0.9022 at 25%; corresponding magnitude-pruning results were 0.9689, 0.9622, and 0.8956. Exact one-sided-axis absorption error was 1.8e-15.

Limitations: Only one fixed-seed digits split and one small MLP were tested. Masks were applied as structured zeroing rather than physically rebuilding compact layers, so parameter/FLOP and wall-clock speedups were not measured. The m_l(epsilon) stopping-rule search, multiple calibration patches/seeds, and CNN/real CIFAR-10 evaluation were not tested.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.08561) Sep 4, 2026

Significantly WORSE than the tuned baseline (+7.3%, p=0.0156, wins 1 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.6
Idea mean: 8.1544
Effect (Δ): 0.5544 (+7.3%; negative = idea better)
Wins: 1 / 8 paired seeds
p-value: 0.0156 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±6.2%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the tabular track and one 50% retained-width setting were tested; physical compact-layer reconstruction and measured wall-clock/FLOP speedups were not benchmarked. The benchmark's quantile criterion is empirical rather than an exact all-input strict sign certificate, and the mechanism signature consequently tests behavior on a finite held-out patch.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 10/10

Built a local stage-2 implementation of task-visible axis pruning for the structurally matched tabular ReLU MLP track, using the benchmark training path, equal learning-rate sweeps, eight paired seeds, and post-training fine-tuning. Magnitude pruning achieved test MSE 7.6000 versus 8.1544 for task-visible pruning, with paired delta +0.5544 and permutation p=0.0156, so the idea was significantly worse. The model-derived mechanism signature also failed: the observed maximum replacement error was 0.1555 rather than the predicted zero, so there is no evidence of the promised effect on this benchmark.

Baseline: Magnitude pruning at 50% hidden width; best lr=0.01; full 8-seed mean test MSE 7.599956, std 0.805135.
Idea: Task-visible quantile crossing pruning at 50% hidden width; best lr=0.01; full 8-seed mean test MSE 8.154385, std 1.038946; paired delta +0.554429; 1/8 wins; permutation p=0.0156.
Mechanism signature: 53 trained-model one-sided selected axes tested; predicted maximum replacement error 0.0, observed maximum error 0.155509, confirmed=false.

Benchmark result