Python crawler Pragmatic series I

Python crawler Pragmatic series I
Python crawler Pragmatic series I
Note:

In this series, I will introduce how to use python to write a crawler and use it to do some practical things. be pragmatic.

Ultimate goal:

Capture the information of the Guarantee Company on the market, analyze the statistics, and save it to excel.

Download the information of each guarantee company on the market, record the company name, contact phone number, service scope and other basic information and save it to Excel.

Objectives:

Learn how to download a web page and perform simple analysis.

Download webpage:

Download the Baidu homepage (not logged on ). For python basic knowledge, click here: get started with Python. For python urllib learning, click here: usage of functions in urllib. For file read/write learning, click here: python file read/write.

Code:

#-*-Coding: UTF-8-*-from urllib import urlretrievedef getWebPage (url): ''' download the webpage to local based on the given url ''' try: '''urlretrieve function will download the webpage targeted by the url to the temporary file ''' revtal = urlretrieve (url) [0] random t IOError: revtal = None if revtal: # If revtal is not empty, it indicates that the webpage has been successfully downloaded, so we can process it with saveWebPage (revtal) def saveWebPage (webpage ): ''' Save the downloaded webpage to the file.txt file ''' f = open (webpage) # open lines = f. readlines () # Save the downloaded webpage information to lines. close () # close the file object fobj = open ("file.txt", 'w' cannot open file.txt in the written format. If file.txt does not exist, a fobj.writelines(lines#write the webpage information to fobj in file.txt. close () # close the object if _ name _ = '_ main _': getWebPage (url = 'HTTP: // www.baidu.com ')

Running result:

Upload a small baidu.txt file of 91kb. the file content is Baidu homepage html code.

Resolution webpage:

To parse a webpage, you must use the python re module, that is, the regular expression module. You can click Regular Expressions for 30 minutes to learn regular expressions. Click the re module function to learn the regular expressions used in python.

There are five links on the Baidu homepage (not logged on), including "news", "hao123", "map", "video", and "post it. We tried to retrieve the values of these five links.

Code:

#-*-Coding: UTF-8-*-import redef getNames (): ''' retrieve the five link values of the Baidu homepage ''' f = open ("baidu.txt ", 'R') # Open the obtained Baidu homepage file lines = f. readlines () # Read the file content to lines f. close () # close the object p = R' \ bclass = "mnav"> (. *?) <'# Regular Expression matching the link value. The brackets are the groups we want to retrieve. # We perform regular expression search for each row. Only one row exists. If the returned list is not empty, for line in lines: if re. findall (p, line) <> []: result = re. findall (p, line) # Let's print it out and see for word in result: print word, if _ name _ = '_ main _': getNames ()

Running result: