1. Goal: Development of lightweight crawlers (excluding login and JavaScript asynchronous Loading)
Do not need to log on static Web page crawl
2. content:
2.1 Crawler Introduction
2.2 Simple Crawler Architecture
2.3 URL Manager
2.4 Web Downloader (urllib2)
2.5 Web Parser (beautifulsoup)
2.6 Full Example: Crawl Baidu Encyclopedia 1000 page data related to Python entry
3. Crawler introduction: a program to automatically crawl Internet information
Crawler value: Internet data, for my use.
4. Simple Crawler Architecture:
Run the Process:
5. URL Manager: Managing the collection of crawled URLs and crawled URLs
-prevents repetitive gripping and prevents cyclic gripping
-implementation method:
6. Web downloader: Download the Web page corresponding to the Internet URL to a local tool
Classification
-urllib2 How to download Web pages:
1. The most concise method: URL ===> urllib2.urlopen (url)
Import urllib2# Direct Request response = Urllib2.urlopen (' http://www.baidu.com ') # Get the status code, if it is 200 means get success print Response.getcode () # Read Content cont = Response.read ()
2. Add data, http header: (url,data,header) ===> urllib2. Request ===> Urllib2.urlopen (request)
Import urllib2# Create request Object request = urllib2. Request (url) # Add Data request.add_data (' a ', ' 1 ') # Add HTTP headerrequest.add_header (' user-agent ', ' mozilla/5.0 ') # Send request Get result response = Urllib2.urlopen (request)
3. Add a processor for a special scenario:
Import urllib2, cookielib# Create cookie Container CJ = cookielib. Cookiejar () # creates 1 Openeropener = Urllib2.build_opener (urllib2. Httpcookieprocessor (cj) # to Urllib2 install Openerurllib2.install_opener (opener) # Use URLLIB2 with cookies to access the page response = Urllib2.urlopen ("http://www.baidu.com/")
7. Urllib2 Instance Code demo:
#-*-coding:utf-8-*-"" "Created on Tue Feb 10:31:06 2017@author:wayne" "" import urllib2, cookieliburl = "http://www.b Aidu.com "print" the 1st method "response1 = Urllib2.urlopen (url) print response1.getcode () print len (response1.read ()) Print "the 2nd method" request = urllib2. Request (url) Request.add_header ("user-agent", "mozilla/5.0") response2 = urllib2.urlopen (request) Print Response2.getcode () print len (response2.read ()) print "the 3rd method" CJ = cookielib. Cookiejar () opener = Urllib2.build_opener (urllib2. Httpcookieprocessor (cj)) response3 = urllib2.urlopen (url) print response3.getcode () print cjprint response3.read ()
8. Web Parser: Tools to extract valuable data from Web pages
Web parser for Python:
Structured Parsing-DOM (Document Object Model) tree:
9. Web Parser-Beautiful Soup
9.1 Beautiful Soup
-Python Third-party Library for extracting data from HTML or XML
-official website: Http://www.crummy.com/software/BeautifulSoup
9.2 Installing and testing the BEAUTIFULSOUP4
-installation: pip Install Beautifulsoup4
-test: Import BS4
9.3 Beautiful Soup Syntax
9.4 Creating a BeautifulSoup Object
From BS4 import beautifulsoup# creating BeautifulSoup objects based on HTML page strings soup = beautifulsoup ( html_doc, # HTML Document string ' H Tml.parser ' # HTML parser from_encoding= ' utf-8 ' # HTML document encoding )
9.5 Search Nodes (find_all, Find)
# method: Find_all (name, attrs, String) # Find all nodes labeled a soup.find_all (' a ') # Find all tags as a, link nodes soup.find_all in/view/123.htm form (' a ' , href= '/view/123.htm ') soup.find_all (' a ', href=re.compiler (r '/view/\d+\.htm ')) # Find all tags as div, class abc, Text for Python node Soup.find_all (' div ', class_= ' abc ', string= ' Python ')
9.6 Accessing node information
# get Node: <a href= ' 1.html ' >python</a># gets the label name of the node to find node.name# gets the href attribute of the A node found node[' href ']# Gets the link text of a node found to Node.get_text ()
BeautifulSoup Instance Test
#-*-coding:utf-8-*-"" "Created on Tue Feb 11:00:42 2017@author:wayne" "" from BS4 import Beautifulsoupimport rehtml_d OC = "" "
Python Development Simple Crawler-basic Framework