The previous articles are mainly about basic knowledge. I guess all my shoes are being moled. do you want to write or not write crawlers ?? Well, this article will write a simple python crawler code for Baidu Post. I will not talk nonsense here. I will directly discuss the code and explain the code in the comments. don't ask me if you don't understand it. learn the basic knowledge!
The code is as follows:
#-*-Coding: UTF-8 -*-
#---------------------------------------
# Program: Baidu Post bar crawler
# Version 0.1
# Author: why
# Date: 2013-05-14
# Programming language: Python 2.7
# Operation: enter a paging address, remove the last number, and set the start and end pages.
# Function: Download all pages on the corresponding page and store them as html files.
#---------------------------------------
Import string, urllib2
# Defining Baidu functions
Def baidu_tieba (url, begin_page, end_page ):
For I in range (begin_page, end_page + 1 ):
SName = string. zfill (I, 5) + '.html '# automatically fill in the six-digit file name
Print 'download' + str (I) + 'web page and store it as' + sName + '......'
F = open (sName, 'W + ')
M = urllib2.urlopen (url + str (I). read ()
F. write (m)
F. close ()
# -------- Enter the parameter ---------------- here ------------------
# This is the address of a post in Baidu Post bar of Shandong University.
# Bdurl = 'http: // tieba.baidu.com/p/2296017831? Pn ='
# IPostBegin = 1
# IPostEnd = 10
Bdurl = str (raw_input (u' enter the address of the post, remove the number "\ n" after pn = '))
Begin_page = int (raw_input (u' enter the start page number: \ n '))
End_page = int (raw_input (u' the number of pages at the end: \ n '))
# -------- Enter the parameter ---------------- here ------------------
# Call
Baidu_tieba (bdurl, begin_page, end_page)
The above is a simple piece of code that python crawls Baidu Post Bar. it is very practical. you can expand it on your own.