# Task-Tangent Capture Pruning

- ID: 2772
- Canonical URL: https://synthcore.org/idea/2772/task-tangent-capture-pruning
- API JSON: https://synthcore.org/api/idea/2772.json
- API Markdown: https://synthcore.org/api/idea/2772.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.27748](https://arxiv.org/abs/2608.27748)
- Category: architecture
- Solves: memory, stability, accuracy
- ML areas: pruning, training-dynamics, fine-tuning
- Math tags: functional-analysis, linear-algebra, dynamical-systems
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Prune parameter directions according to how much task-relevant Jacobian energy they carry, rather than by weight magnitude or individual gradient magnitude. Keep a mask whose discarded tangent component is at most an empirical fraction epsilon of the full tangent vector for calibration task directions, thereby preserving the local output dynamics seen by the task.

## Mathematical statement

The paper's relative task-capture condition is \(\|(I-P)T^{*}u\|_{H}\leq\epsilon\|T^{*}u\|_{H}\). Here \(H\) is the parameter Hilbert space with its Euclidean inner product, \(T:H\to Y\) is the network parameter Jacobian mapping parameter perturbations to output perturbations, \(T^{*}:Y\to H\) is its adjoint, \(u\in Y\) is a task-space output direction, and \(P:H\to H\) is an orthogonal pruning projection that retains the unpruned parameter coordinates. The left side is the norm of the parameter update direction discarded by pruning, while the right side is the total parameter tangent direction induced by \(u\). The condition is imposed for task-relevant \(u\), such as residuals, label directions, or a low-dimensional basis of the cyclic task space generated by the output-space operator \(G=TT^{*}\) and task subspace \(M\). For a finite calibration set with output-direction matrix \(U=[u_1,\ldots,u_m]\), use the empirical squared ratio \(\widehat r(P)^2=\frac{\sum_i\|(I-P)T^{*}u_i\|_2^2}{\sum_i\|T^{*}u_i\|_2^2}\leq\epsilon^2\). The ML adaptation uses \(T_P=TP\) and compares the full local kernel \(G=TT^{*}\) with the pruned kernel \(G_P=T_PT_P^{*}=TPT^{*}\); low tangent loss is expected to preserve short-time linearized training dynamics.

## Key formulas

- $$\|(I-P)T^{*}u\|_{H}\leq\epsilon\|T^{*}u\|_{H}$$
- $$\widehat r(P)^2=\frac{\sum_{i=1}^{m}\|(I-P)T^{*}u_i\|_2^2}{\sum_{i=1}^{m}\|T^{*}u_i\|_2^2}\leq\epsilon^2$$
- $$G=TT^{*},\qquad T_P=TP,\qquad G_P=T_PT_P^{*}=TPT^{*}$$
- $$\dot z(t)=-Gz(t)\quad\longrightarrow\quad\dot z_P(t)=-G_Pz_P(t)$$

## Implementation notes

Integrate this into structured or unstructured weight pruning after a short calibration phase, using the parameter Jacobian of network outputs on a small representative dataset. (1) Exact integration point: choose a candidate binary parameter mask \(P\) for one layer or the whole model; take the Jacobian from output logits or regression outputs with respect to candidate weights, and make pruning decisions before the main fine-tuning run. Do not form the full Jacobian: use reverse-mode vector-Jacobian products to compute \(g_i=T^*u_i\) for each calibration direction. (2) Pseudocode: collect calibration examples \((x_i,y_i)\); compute outputs \(f_\theta(x_i)\); set \(u_i=f_\theta(x_i)-y_i\), or use several random output-space directions for classification; compute \(g_i=T^*u_i\); initialize a mask with magnitude pruning or all weights retained; for each candidate block or weight set \(S\), score removal by \(q(S)=\sum_i\|g_i[S]\|_2^2/\sum_i\|g_i\|_2^2\); greedily remove the lowest-score candidates while \(q(S)\leq\epsilon^2\), setting \(P_{jj}=0\) for removed coordinates and one otherwise; then fine-tune the masked network. (3) The mathematical quantity is the discarded-to-total tangent norm ratio; task directions, mask granularity, and greedy approximation are engineering choices. Estimate the ratio on held-out calibration batches and optionally refresh it after each fine-tuning epoch. For a dynamics diagnostic, estimate kernel-vector products and monitor \(\|(G-G_P)v\|/\|Gv\|\) for random output vectors \(v\). (4) First experiment: use a ResNet-20 on CIFAR-10 or a 6-layer Transformer on a small language-modeling corpus, comparing magnitude pruning, gradient pruning, and tangent-capture pruning at 50% and 80% sparsity. Test \(\epsilon\in\{0.05,0.1,0.2\}\). Measure immediate validation loss increase, fine-tuning loss descent at equal optimizer steps, final accuracy or perplexity, and wall-clock and memory reduction with a genuinely sparse kernel. Success means a smaller immediate loss shock and faster recovery than magnitude pruning at equal sparsity, while the held-out empirical ratio stays below the selected epsilon.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a reproducible Task-Tangent Capture Pruning MVP with epsilon-controlled tangent-energy masks and a small masked-regression comparison against magnitude pruning. The mechanism manifested: discarded tangent ratios stayed below every epsilon target, the squared-energy identity matched to 1.1e-16, and tangent ranking discarded less task energy at all tested sparsities. In the toy regression, tangent pruning achieved lower final loss after 30 steps at both 50% and 80% pruning, although its immediate loss shock was larger in this single run.

### Mechanism check

- Verdict: Built a reproducible Task-Tangent Capture Pruning MVP with epsilon-controlled tangent-energy masks and a small masked-regression comparison against magnitude pruning. The mechanism manifested: discarded tangent ratios stayed below every epsilon target, the squared-energy identity matched to 1.1e-16, and tangent ranking discarded less task energy at all tested sparsities. In the toy regression, tangent pruning achieved lower final loss after 30 steps at both 50% and 80% pruning, although its immediate loss shock was larger in this single run.
- Confidence: 8/10
- Limitations: This was a toy random-operator verification plus one tiny tanh regression and one seed. It did not test ResNet/CIFAR, gradient pruning, held-out calibration generalization, mask refresh, genuinely sparse kernels, wall-clock speed, or memory reduction.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.4521
- Paired wins: 1/8
- Benchmark verdict: no measurable effect

## Artifacts

- [REPORT.md](https://synthcore.org/code/1093/REPORT.md)
- [bench_report.json](https://synthcore.org/code/1093/bench_report.json)
- [experiment.py](https://synthcore.org/code/1093/experiment.py)
- [report.md](https://synthcore.org/code/1093/report.md)
- [report_bench_2026-09-02T005553.md](https://synthcore.org/code/1093/report_bench_2026-09-02T005553.md)
- [results.json](https://synthcore.org/code/1093/results.json)
- [stage2_bench.py](https://synthcore.org/code/1093/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1093)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.