# Dual-consistent feature–sample pruning

- ID: 3053
- Canonical URL: https://synthcore.org/idea/3053/dual-consistent-feature-sample-pruning
- API JSON: https://synthcore.org/api/idea/3053.json
- API Markdown: https://synthcore.org/api/idea/3053.md
- Verification status: mechanism_failed
- Source: [arXiv:2609.02418](https://arxiv.org/abs/2609.02418)
- Category: memory
- Solves: memory, speedup, sample-efficiency
- ML areas: pruning, training, optimizer
- Math tags: convex-analysis, optimization, linear-algebra
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Train a convex surrogate attached to a neural network, such as a sparse linear probe or adapter, and use its primal and Fenchel-dual representations to produce mutually consistent feature and sample masks. The same screening operation can be evaluated on columns of the primal design matrix or rows of the dual representation, allowing data and feature reduction without independently tuned heuristics.

## Mathematical statement

The paper defines an FR representation p=(f,g,A,a) for p(x)=f(Ax)+g(x)+a, where A∈R^{m×n}, x∈R^n is the parameter vector, f:R^m→R∪{+∞} and g:R^n→R∪{+∞} are separable proper closed convex functions, and a∈R is constant. The Fenchel–Rockafellar dual is d(y)=f*(y)+g*(-A^T y)+a over y∈R^m, with dual variable y and convex conjugates f*(y)=sup_z{y^Tz−f(z)} and g*(u)=sup_x{u^Tx−g(x)}. Matrix columns represent features and rows represent samples. The paper proves the equivariance relation D∘F=S∘D: dualization D followed by feature screening F equals sample screening S followed by dualization. For g(x)=λ||x||_1, λ>0, the KKT condition gives x_j*=0 whenever |A_{:j}^T y*|<λ, where A_{:j} is feature column j and y* is an optimal dual point. With an approximate dual point y and safe radius R, use the conservative test |A_{:j}^T y|+R||A_{:j}||_2<λ. Sample scores can be obtained from separable dual contributions f_i*(y_i), but sample deletion is heuristic unless a corresponding safe bound is established for the chosen loss.

## Key formulas

- $$\min_{x\in\mathbb{R}^{n}}p(x)=f(Ax)+g(x)+a,\qquad A\in\mathbb{R}^{m\times n},\quad a\in\mathbb{R},$$
- $$\min_{y\in\mathbb{R}^{m}}d(y)=f^*(y)+g^*(-A^T y)+a,\qquad f^*(y)=\sup_z\{y^Tz-f(z)\},\quad g^*(u)=\sup_x\{u^Tx-g(x)\},$$
- $$g(x)=\lambda\lVert x\rVert_1\quad\Longrightarrow\quad x_j^*=0\ \text{whenever}\ \left|A_{:j}^{T}y^*\right|<\lambda,$$
- $$D\!\left(F(f,g,A,a)\right)=S\!\left(D(f,g,A,a)\right).$$

## Implementation notes

Integrate the method into a convex subproblem: freeze an encoder, form embeddings h_i, and train a sparse linear probe or adapter w with L(w)=Σ_i f_i((Aw)_i)+λ||w||_1, where row i of A is the embedding of sample i and column j is an embedding coordinate or adapter feature. At each pruning round: (1) warm-start the convex head and run proximal-gradient updates; (2) obtain a dual estimate y from the residual or loss-gradient map and project it onto the known dual-feasible set; (3) calculate feature scores c_j=|A_{:j}^T y|; if a primal–dual gap gives radius R, permanently remove feature j only when c_j+R||A_{:j}||_2<λ; otherwise rank features by c_j and remove the lowest-scoring fraction as an explicitly heuristic variant; (4) calculate sample scores r_i=f_i*(y_i)−min_t f_i*(t), normalize them by loss scale, and remove the lowest-scoring examples subject to a validation safeguard; (5) apply both masks to A and continue warm-started optimization. The mathematical components are the FR dual, the KKT feature rule, the safe-radius correction, and the row/column equivariance. Estimate R from the current primal–dual gap and the local strong-convexity or Lipschitz constants; estimate sample thresholds empirically. First test CIFAR-10 with a frozen ResNet-18 and sparse logistic-regression head. Compare magnitude feature pruning plus random sample pruning against dual-consistent pruning at 25%, 50%, and 75% reduction. Record head FLOPs, retained feature dimension, retained samples, primal–dual gap, convergence speed, and validation accuracy. A positive result is equal accuracy with fewer head FLOPs and examples, or faster loss descent at equal accuracy; then transfer selected embedding/channel masks to full-network fine-tuning.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a sparse least-squares FR/L1 prototype with dual-feasibility projection, safe-radius feature screening, dual feature ranking, and separable dual sample scoring. The mathematical checks succeeded: 15/15 safe-screened zero features, no false safe eliminations, KKT error 2.5e-10, and exact column/row mask equivariance. However, the pruning strategy did not provide a reliable validation win: at 50% and 75% reduction its validation MSE was much worse than magnitude-feature plus random-sample pruning, so the claimed practical benefit was not observed.

### Mechanism check

- Verdict: Built a sparse least-squares FR/L1 prototype with dual-feasibility projection, safe-radius feature screening, dual feature ranking, and separable dual sample scoring. The mathematical checks succeeded: 15/15 safe-screened zero features, no false safe eliminations, KKT error 2.5e-10, and exact column/row mask equivariance. However, the pruning strategy did not provide a reliable validation win: at 50% and 75% reduction its validation MSE was much worse than magnitude-feature plus random-sample pruning, so the claimed practical benefit was not observed.
- Confidence: 9/10
- Limitations: This is a synthetic squared-loss convex-head experiment, not CIFAR-10 or a frozen ResNet. The sample-removal score is heuristic and no safe sample-deletion bound or neural-network fine-tuning transfer was tested; only one deterministic dataset/seed was evaluated.

## Artifacts

- [report.md](https://synthcore.org/code/1227/report.md)
- [run_experiment.py](https://synthcore.org/code/1227/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/1227)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.