Python Basics and Crawler technology
Course Study Address: http://www.xuetuwuyou.com/course/195
The course out of self-study, worry-free network: http://www.xuetuwuyou.com
Course Introduction
As a convenient way to collect information from the Internet and extract available information from it, web crawler technology becomes more and more useful. With a simple programming language like Python, you can crawl complex sites with a few programming skills. Python Basic and Crawler technology explains how to crawl data from static pages and how to manage server load using caching. In addition, this course describes how to crawl data using AJAX URLs and Firebug extensions, as well as more facts about crawling techniques, such as using browser rendering, managing cookies, extracting data from a complex site protected by a validation code by submitting a form, and more.
This course uses Scrapy to create an advanced web crawler and to crawl some of the real web sites.
Python Basic and Crawler technology describes the following:
Crawl sites by tracking links;
Extracting data from the page using XML;
Build a thread crawler to crawl pages in parallel;
Cache the downloaded content to reduce bandwidth consumption;
Parsing web sites that rely on JavaScript;
Interacting with forms and sessions;
Solve the problem of verification code for protected pages;
Reverse engineer the AJAX call;
Use Scrapy to create advanced crawlers.
Course Catalogue
1th: Introduction to Python Installation and idle tools and syntax
1. Course Introduction
2.Python Introduction, Installation and idle tools _ what are programming languages and Python history
3.Python Introduction, Installation and Idle tool _python features
4.Python Introduction, Installation and idle Tools _python development environment Configuration-1
5.Python Introduction, Installation and idle Tools _python development environment Configuration-2
6.Python Introduction, Installation and idle Tools _ this chapter work
Getting Started with 7.Python syntax basic data types _number and float and none and bool data types
Getting Started with 8.Python syntax basic data type _string type
9.Python syntax Getting Started _ variables
The 2nd chapter: Python Data structure and control flow statements
10. Basic program composition and input and output
11. Common methods for converting between basic data types
12.Python Data Structure-list
13.Python Data Structures-Collections
14.Python Data Structure-tuples
15.Python Data Structure-dictionary
16.Python Operators and expressions
Simple if statement of 17.Python conditional statement
Multiple conditional if statements for 18.Python conditional statements
Complex conditions and nested IF statements for 19.Python conditional statements
The 3rd Chapter: Introduction to Python crawler technology
Import of 20.Python loop statements
The while loop of the 21.Python Loop statement
For loop of 22.Python Loop statement
Loop nesting of 23.Python loop statements
Circular control statements for 24.Python loop statements
Crawler base of 25.Python crawler
26.Python Crawler Crawler Introduction of the principle of the article
27.Python Crawler Crawler Introduction to the Request object
28.Python Crawler crawler Beginner's small case crawl simple Web page
Python Web crawler Practical Tutorial (full version)
Course Study Address: http://www.xuetuwuyou.com/course/168
The course out of self-study, worry-free network: http://www.xuetuwuyou.com
Web crawler Technology Introduction _python Foundation and crawler Technology
