Python+urllib+beautifulsoup implementation of a simple crawler

Urllib is a library of the URL of the sequence of operations provided in python3.x, which can easily impersonate a user to access a Web page using a browser.

Beautiful Soup is a python library that extracts data from HTML or XML files. It is able to use your favorite converter to achieve idiomatic document navigation, find, modify the way the document. Beautiful Soup will save you hours or even days of working hours.

1. Install Python's latest installation package 3.5.2

　　: https://www.python.org/

Note: When installing, note the option to add to path (System environment variable)

Verify that the installation was successful: Enter pythonin command-line CMD and the installation succeeds if you enter the editing environment for Python.

　　

2, Installation Urllib

Enter in the python command line: fromurllib.request import Urlopen

Appears as follows: The installation was successful without an error message

　　

3, Installation BeautifulSoup

Enter in cmd:pip install beautifulsoup4

　　

Verify that the installation succeeds:form BS4 Import beautifulsoup If no error is installed

　　

4. Introduction to the usage of urllib

(1) Steps to use

• Request Module Import urllib Library from urllib import request
• Requested URL req = Request. Request (URL)
• Set Request Header Req.add_header (Key,value)
• Output data using the Response object print (Resp.read (). Decode (' Utf-8 '))

 from  urllib import   Requestreq  = Request. Request ( http://www.baidu.com   " ) Req.add_header ( "  user-agent  , "  mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) gecko/20100101 firefox/48.0   " ) Resp  = Request.urlopen (req)  print  (Resp.read (). Decode ( utf-8  Span style= "color: #800000;" >"))

Here I use sublime Text test: Can see Baidu homepage data is can be obtained down.

　　

(2) Post request

• Import urllib parse:from urllib Import Parse
• Using UrlEncode to generate post data

PostData = Parse.urlencode ([

(KEY1,VAL1),

(Key2,val2),

(KEY3,VAL3)

])

• Send POST request Request.urlopen with postdata (Req,data = Postdata.encode (' utf-8 '))
• Get Request Status Resp.status
• Get the server type Resp.reason

 fromUrllib.requestImportUrlopen fromUrllib.requestImportRequest fromUrllibImportParsereq= Request ('http://huochepiao.114piaowu.com/train/ydTrainZdz_searchAdapter.action') PostData=Parse.urlencode ({'fromstation':'Shanghai',    'Godatestr':'2016-09-07',    'SearchType': 0,'tostation':'Guangzhou'}). Encode ('Utf-8') Req.add_header ("user-agent","mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) gecko/20100101 firefox/32.0") Resp= Urlopen (Req,data =postdata)Print(Resp.read (). Decode ('Utf-8'))

5, BeautifulSoup Use introduction

http://beautifulsoup.readthedocs.io/zh_CN/latest/#id20

Python+urllib+beautifulsoup implementation of a simple crawler