The practice of Python crawler

Some spit Slots

The first time the real start to write a technical blog, always think of their technology is not good, not to write the requirements of the blog, and then do not dare to write, and then I found that the road is endless technology, you can not learn all things, all the people are in the exchange of growth, So today decided to come up with some dry goods to share with you.

This topic is called Python crawler best practice, first of all to say why to write crawler-related. Because I like the language of Python, simple, powerful and outrageous, it is also very easy to get started. And people mention that Python is always the first to think of reptiles, so I decided to share what I know knowledge of reptiles to everyone. As for why the name of best practice, because I was from a pure small white slowly grew up. I think we all have the same experience, is that whenever you meet a technical knowledge of their own point of interest is a very systematic foundation of the introductory tutorial, so that they really go into this field, but unfortunately, the technical blog is always so advanced, so that those who do not have the basis of the small white people wandering in the infinite pain, Want to see and do not understand. For the great gods, this of course is good. But it's too unfriendly for small white people who are good at it, but who don't know the industry.

Best Practice Process

I was very painful when I first learned the crawler, because there is no systematic tutorial, only one article to see some scattered blog to learn. So I don't want a little white people like me have the same experience, based on my own experience, summed up my set of best practice flow:

1. Configure the desired environment (PS: There is always a lot of small white in this.)

2. Deep understanding of Demo

3. Follow the demo for your own practice

4. Self-discovery and expansion of content to achieve their goals

We need to learn more than programming technology, including problem-solving mode of thinking, which is the focus of our study.
PS: I am not a big God, so if there are objections to the above process, everyone has their own way of learning.

Practice Content

Down is the practical content of the topic Related:

• Crawler-related knowledge, including the use of basic crawlers and frameworks like Pyspider

• Advanced crawlers, including using selenium to simulate users, using multiple processes in crawlers

• Simple knowledge of data processing, because a lot of people don't know what to do with data

• Use some cartographic plugins to display statistical data in a graphical way

• Simple Django Build station knowledge (how to show the data)

PS: This blog does not tell the relevant knowledge of Python, if there are shoes with a python grammar or something still do not know, recommend to see Liaoche python
So after reading this topic, you should master how to write a crawler, how to use the crawler framework, how to do simple data analysis and statistics, how to make a chart based on statistical information, how to display their own charts through the site. This is our ultimate goal.