Python Development Simple Crawler-basic Framework

Source: Internet
Author: User

1. Goal: Development of lightweight crawlers (excluding login and JavaScript asynchronous Loading)

Do not need to log on static Web page crawl

2. content:

2.1 Crawler Introduction

2.2 Simple Crawler Architecture

2.3 URL Manager

2.4 Web Downloader (urllib2)

2.5 Web Parser (beautifulsoup)

2.6 Full Example: Crawl Baidu Encyclopedia 1000 page data related to Python entry

3. Crawler introduction: a program to automatically crawl Internet information

  

Crawler value: Internet data, for my use.

  

4. Simple Crawler Architecture:

  

Run the Process:

  

5. URL Manager: Managing the collection of crawled URLs and crawled URLs

-prevents repetitive gripping and prevents cyclic gripping

  

-implementation method:

  

6. Web downloader: Download the Web page corresponding to the Internet URL to a local tool

  

Classification

  

-urllib2 How to download Web pages:

1. The most concise method: URL ===> urllib2.urlopen (url)

Import urllib2# Direct Request response = Urllib2.urlopen (' http://www.baidu.com ') # Get the status code, if it is 200 means get success print Response.getcode () # Read Content cont = Response.read ()

2. Add data, http header: (url,data,header) ===> urllib2. Request ===> Urllib2.urlopen (request)

Import urllib2# Create request Object request = urllib2. Request (url) # Add Data request.add_data (' a ', ' 1 ') # Add HTTP headerrequest.add_header (' user-agent ', ' mozilla/5.0 ') # Send request Get result response = Urllib2.urlopen (request)

3. Add a processor for a special scenario:

Import urllib2, cookielib# Create cookie Container CJ = cookielib. Cookiejar () # creates 1 Openeropener = Urllib2.build_opener (urllib2. Httpcookieprocessor (cj) # to Urllib2 install Openerurllib2.install_opener (opener) # Use URLLIB2 with cookies to access the page response = Urllib2.urlopen ("http://www.baidu.com/")

7. Urllib2 Instance Code demo:

#-*-coding:utf-8-*-"" "Created on Tue Feb 10:31:06 2017@author:wayne" "" import urllib2, cookieliburl = "http://www.b Aidu.com "print" the 1st method "response1 = Urllib2.urlopen (url) print response1.getcode () print len (response1.read ()) Print "the 2nd method" request = urllib2. Request (url) Request.add_header ("user-agent", "mozilla/5.0") response2 = urllib2.urlopen (request) Print Response2.getcode () print len (response2.read ()) print "the 3rd method" CJ = cookielib. Cookiejar () opener = Urllib2.build_opener (urllib2. Httpcookieprocessor (cj)) response3 = urllib2.urlopen (url) print response3.getcode () print cjprint response3.read ()

8. Web Parser: Tools to extract valuable data from Web pages

  

Web parser for Python:

  

Structured Parsing-DOM (Document Object Model) tree:

  

9. Web Parser-Beautiful Soup

9.1 Beautiful Soup

-Python Third-party Library for extracting data from HTML or XML

-official website: Http://www.crummy.com/software/BeautifulSoup

9.2 Installing and testing the BEAUTIFULSOUP4

-installation: pip Install Beautifulsoup4

-test: Import BS4

9.3 Beautiful Soup Syntax

    

    

9.4 Creating a BeautifulSoup Object

From BS4 import beautifulsoup# creating BeautifulSoup objects based on HTML page strings soup = beautifulsoup (                     html_doc,                     # HTML Document string                     ' H Tml.parser '                  # HTML parser                     from_encoding= ' utf-8 '     # HTML document encoding                     )

9.5 Search Nodes (find_all, Find)

# method: Find_all (name, attrs, String) # Find all nodes labeled a soup.find_all (' a ') # Find all tags as a, link nodes soup.find_all in/view/123.htm form (' a ' , href= '/view/123.htm ') soup.find_all (' a ', href=re.compiler (r '/view/\d+\.htm ')) # Find all tags as div, class abc, Text for Python node Soup.find_all (' div ', class_= ' abc ', string= ' Python ')

9.6 Accessing node information

# get Node: <a href= ' 1.html ' >python</a># gets the label name of the node to find node.name# gets the href attribute of the A node found node[' href ']# Gets the link text of a node found to Node.get_text ()

BeautifulSoup Instance Test

#-*-coding:utf-8-*-"" "Created on Tue Feb 11:00:42 2017@author:wayne" "" from BS4 import Beautifulsoupimport rehtml_d OC = "" "

    

  

Python Development Simple Crawler-basic Framework

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.