Scrapy crawler -- 01

Scrapy is a fast, high-level screen capture and web capture framework developed by python. It is used to capture web sites and extract structured data from pages.

-- From Wiki

To put it bluntly, it is a python-based crawler framework.

Installation:

• Ubuntu 14.04

• Python2.7 (not supported by python3, not lazy by the author. It is because the scrapy framework has not completely migrated twisted to python3)

• Pip
  

sudo pip2 install scrapy

Note: Although pip3 can also be installed with scrapy, it cannot be used because of the lack of support libraries... Hey, python2.

Usage:

1. Create a Project Test

scrapy startproject tutoria

This creates a directory structure:

tutorial/    scrapy.cfg    tutorial/        __init__.py        items.py        pipelines.py        settings.py        spiders/            __init__.py            ..

The official website is explained as follows:

• Scrapy. cfg: The project configuration file (project configuration file)

• Tutorial/: the project's Python module, You'll later import your code from here. (I don't know how to translate the custom part of the project)

• Tutorial/items. py: the project's items file. (the project's items file is actually the structure definition of the data to be captured)

• Tutorial/pipelines. PY: the project's pipelines file. (The pipelines file of the project can be defined here to export the captured data. Pip has the scrapy-MongoDB pipelines and can directly export the captured data to the pipeline .)

• Tutorial/settings. py: the project's settings file. (project configuration file)

• Tutorial/spiders/: A directory where you'll later put your spiders)

To be continued...

Scrapy crawler -- 01