Scrapy Common Command line tool introduction

View All Commands

Scrapy-h

View Help information

Scapy--help

View version Information

(venv) ql@ql:~$ scrapy versionscrapy 1.1.2 (venv) ql@ql:~$ (venv) ql@ql:~$ scrapy version-vscrapy    : 1.1.2lxml      : 3.6.4.0LIBXML2   : 2.9.4Twisted   : 16.4.0Python:    2.7.12 (Default, Jul  1, 15:12:24)-[GCC 5.4.0 201606 09]pyopenssl:16.1.0 (OpenSSL 1.0.2g-fips  1 Mar) Platform  : linux-4.4.0-36-generic-x86_64- With-ubuntu-16.04-xenial (venv) ql@ql:~$

Create a new project

Scrapy Startproject Spider_name

Build crawler Genspider (Generator spider)

Multiple spiders can exist in a project, but the name must be unique

Scrapy Genspider name domain# such as: #scrapy genspider Sohu sohu.org

See how many crawlers are in the current project

Scrapy List

View opens a Web page using a browser

Scrapy View www.baidu.com

Shell command, enter Scrpay Interactive environment

#进入该url的交互环境scrapy Shell www.dmoz.org/Computers/Programming/Languages/Python/Books/

Then we enter the interactive environment
We mainly use the response command in this, for example, you can use

Response.xpath () #括号里直接加xpath路径

The Runspider command is used to run the created crawler directly and does not run the entire project

Scrapy Runspider crawler Name