The Python crawler engineer with a monthly salary of 45 K tells you how to learn crawlers. It's too detailed !, 45 kpython
If you want to use Python as a crawler, but you still don't know Python, these basics are essential. Many friends, especially students at school, feel this amazing when they come into contact with crawlers. I want to learn it. But I don't know where to start. I am confused and learn a lot. Let's take a look at what I need to learn when I want to use python for crawlers!
Getting started with Python
1. Basic knowledge, data types, and process control.
Life is short, and Python is easy to use
2. Functions
Life is short, just use Python
3. Modules
4. I/O Basics
Life is not short, but Python is also used
5. Linux Basics
Linux Basics
Finally, you need to know more about package management and virtual environments. What about these basics!
Web Front-end
Of course, the basics of optics are far from enough. We need to go deep into crawlers. First, we need to understand how this is done and how it works, so the web Front-end learning is essential. Let's take a look at it.
1. Labels
Tag
2. Style
Style
3. JS syntax
JS syntax
4. jQuery
JQuery
5. Ajax
Ajax
6. bootstrap
Bootstrap
The front-end part does not need to be too deep, but it is enough to learn crawlers. As the saying goes, "to do a good job, you must first sharpen the tool." So after learning the following Python knowledge, you can learn crawlers without any worries.
Python advanced
1. Object-oriented
Object-oriented
2. decorator and inertia
Decorator and inertia
3. Advanced Module
Advanced Module
4. Concurrent Programming (multithreading and multi-process)
What are processes and threads?
Multi-Process
Multithreading
Future Implementation of multi-process and multi-thread
5. Concurrent Programming (coroutine and asynchronous IO)
Coroutine, asynchronous IO
Well, after all these things are learned, you can start official crawler learning. Many things will be easy to understand as soon as you see them, and you will not be able to understand what you want.
Crawler Basics
To learn about crawlers, we must first understand what crawlers are and how they work. Only by knowing ourselves and ourselves can we win a hundred battles. Of course, we all know this. Let's take a look at the basic crawler knowledge.
1. Request and response
Request and response
2. crawler and anti-Crawler
Crawler and anti-Crawler
3. Development Tools
Development Tools
4. Detailed use of the Urllib library and project practices
Urllib library Usage Details and project practices
5. Installation and Use of the requests library and Project Practice
Installation and use of the requests library and Project Practice
6. Too many pictures
Last crawler bullet
Well, if you just want to get into the door, these are almost the same, but I think every crawler partner wants to become a great crawler to attack the city lion! Let's continue to look at advanced learning methods.
Advanced Crawler
1. crawler Framework Implementation
Crawler Framework Implementation
2. Anti-crawling
Anti-crawling Technology
3. Proxy pool implementation
Proxy pool implementation
4. Simulate Login
Simulated Login
5. pyspider framework
Pyspider framework
Advanced Crawler
1. APP crawling
APP crawling
2. Scrapy framework
Scrapy framework
3. Distributed crawler practice
Distributed crawler practice
4. Distributed crawler deployment
Distributed crawler deployment
After learning this knowledge, the next crawler is you!