What is a reptile? A detailed description of the crawler

Hello everyone, recently Bo Master in the study of Python, during the study also encountered some problems, gained some experience, this will be their own learning system to organize down, if you are interested in learning Crawler, you can use these articles as a reference, but also welcome everyone to share learning experience.

Python Version: 2.7,python 3 Please find another blog post.

First, what is a reptile?

Web crawler (also known as Web spider, Network robot, in the middle of the foaf community, more often called the Web Chaser), is a certain rules, automatically crawl the World Wide Web information program or script.

According to my experience, to learn the Python crawler, we have to learn the following points:

Python Basics

Usage of urllib and URLLIB2 libraries in Python

Python Regular Expressions

Python crawler Frame Scrapy

More advanced features of Python crawlers

1.Python Basic Learning

First of all, we want to use Python to write crawler, definitely want to understand the basis of Python, towering high-rise, can not forget the foundation, haha, then I would like to share some of the python I have seen some tutorials, small partners can be used as a reference.

1) MU-class Network Python tutorial

There have been some basic grammar in the MU class on the Internet to see, the above with some exercises, after learning can be used as exercise, feeling the effect is pretty good, but a little regret is the content is basically the most basic, the beginning of the word, this bar

Learning Web site: MU-class Network Python tutorial

2) Liaoche python tutorial

Later, I found that teacher Liao's Python tutorial, which is very easy to understand, it is also very good, if you want to learn more about Python look at this bar.

Study site: Liaoche python Tutorial

3) Concise Python tutorial

There's also a concise Python tutorial that I've seen, and I feel good about it.

Learning Web site: A Concise python tutorial

4) Wanghai's laboratory

This is my undergraduate laboratory seniors, when the introduction of his article, he re-made a summary, and then these series of articles on his basis added some content.

Study site: Wanghai's lab

2.Python urllib and Urllib2 library usage

Urllib and Urllib2 Library is the most basic library of learning Python crawler, using this library we can get the content of the Web page, and the content with regular expression to extract the analysis, to get the results we want. I will share this with you in the course of learning.

3.Python Regular Expressions

A python regular expression is a powerful weapon for matching strings. Its design idea is to use a descriptive language to define a rule for a string, and any string that conforms to the rule, we think it "matches", otherwise the string is illegal. This will be shared at the back of the post.

4. Reptile Frame Scrapy

If you are a Python master, basic crawler knowledge has been mastered, then look for the Python framework, I chose the framework is the Scrapy framework. What are the powerful features of this framework? Here is an official introduction to it:

HTML, built-in support for XML source data selection and Extraction
provides a series of reusable filters (that is, Item loaders) shared between spiders, providing built-in support for intelligent processing of crawling data. The
provides multi-format (JSON, CSV, XML) with feed export, and built-in support for multiple storage backend (FTP, S3, local file system)
provides media pipeline to automatically download pictures (or other resources) from crawled data.
High Extensibility. You can customize your functionality by using signals, a well-designed API (middleware, extensions, pipelines). The
built-in middleware and extensions provide support for the following features:
Cookies and session processing
HTTP Compression
HTTP Authentication
HTTP cache
User-agent impersonation
Robots.txt
Crawl Depth Limit
provides automatic detection and robust encoding support for nonstandard or incorrect encoding declarations in non-English language languages. The
supports the creation of crawlers based on templates. While accelerating the creation of crawlers, keeping the code in large projects more consistent. For more information, see the Genspider command. The
provides a scalable state-gathering tool for multi-crawler performance evaluation and failure detection.
provides an interactive shell terminal for you to test XPath expressions, writing and debugging crawlers provides great convenience
provides System service, simplifies deployment in production environments and runs
built-in WEB service, allowing you to monitor and control your machine
Built-in Telnet terminal, which allows you to view and debug crawlers
Logging provides a convenient way to catch errors during crawling by hooking into a python terminal in the scrapy process
Support Sitemaps crawling
with a cached DNS resolver