Baidu paste the reptile production and embarrassing hundred of the reptile production principle is basically the same, all by viewing the source key data deducted, and then stored to a local TXT file.
Project content:
Written in Python, Baidu paste the Web crawler.
How to use:
After you create a new bugbaidu.py file, and then copy the code inside, double-click Run.
Program function:
Paste the content of the landlord posted txt stored on-premises.
Explanation of principle:
First of all, first glance at a bar, click on the landlord and click on the second page after the URL has changed a little, became:
Http://tieba.baidu.com/p/2296712428?see_lz=1&pn=1
Can be seen, see_lz=1 is only to see the landlord, Pn=1 is the corresponding page number, remember this for future preparation.
This is the URL we need to use.
Next is to view the source of the page.
The first thing to do is to pick out the problem and store it.
Can see Baidu using GBK encoding, the title uses H1 tag:
Copy the Code code as follows:
"Original" fashion chief (about fashion, fame, career, love, inspirational)
Similarly, the body part uses the DIV and class synthetic tags, the next thing to do is to use regular expressions to match.
Run:
Generated TXT file:
Copy the Code code as follows:
#-*-Coding:utf-8-*-
#---------------------------------------
# program: Baidu Paste Stick Crawler
# version: 0.5
# Author: why
# Date: 2013-05-16
# language: Python 2.7
# action: Enter the URL to automatically see only the landlord and save to the local file
# function: Store the contents of the landlord post to the local TXT storage.
#---------------------------------------
Import string
Import Urllib2
Import re
#-----------handle various labels on the page-----------
Class Html_tool:
# use non-greedy mode to match \ t or \ n or a space or hyperlink or picture
Bgnchartononerex = Re.compile ("(\t|\n| | |)")
# match any <> tag with non-greedy mode
Endchartononerex = Re.compile ("<.*?>")
# use non-greedy mode to match any
Label
Bgnpartrex = Re.compile (" ")
Chartonewlinerex = Re.compile ("(
|
|||)")
Chartonexttabrex = Re.compile ("")
# turn some of the HTML symbol entities into original symbols
Replacetab = [("<", "<"), (">", ">"), ("&", "&"), ("&", "\" "), (" "," ")]
def Replace_char (self,x):
x = self. Bgnchartononerex.sub ("", X)
x = self. Bgnpartrex.sub ("\ n", X)
x = self. Chartonewlinerex.sub ("\ n", X)
x = self. Chartonexttabrex.sub ("\ T", X)
x = self. Endchartononerex.sub ("", X)
For T in Self.replacetab:
x = X.replace (t[0],t[1])
return x
Class Baidu_spider:
# Declaration of related properties
def __init__ (Self,url):
Self.myurl = URL + '? see_lz=1 '
Self.datas = []
Self.mytool = Html_tool ()
Print U ' has started Baidu paste bar crawler, clicks
# Initialize Load page and transcode it to save
def Baidu_tieba (self):
# Read the original information of the page and transcode it from GBK
MyPage = Urllib2.urlopen (Self.myurl). Read (). Decode ("GBK")
# Calculate how many pages the landlord published content
EndPage = Self.page_counter (mypage)
# get the title of the thread
title = Self.find_title (MyPage)
Print U ' article name: ' + title
# Get the final data
Self.save_data (Self.myurl,title,endpage)
#用来计算一共有多少页
def page_counter (self,mypage):
# matches "Total APage "To get a total of how many pages
Mymatch = Re.search (R ' class= "Red" > (\d+?)', MyPage, re. S
If Mymatch:
endpage = Int (Mymatch.group (1))
Print U ' crawler report: Found landlord total%d pages of original content '% EndPage
Else
EndPage = 0
Print U ' bot report: Unable to calculate how many pages the landlord published content! '
return EndPage
# used to find the title of the Post
def find_title (self,mypage):
# match
Xxxxxxxxxx
Find the title
Mymatch = Re.search (R ' (.*?)', MyPage, re. S
title = U ' No title '
If Mymatch:
title = Mymatch.group (1)
Else
Print U ' crawler report: Unable to load article title! '
# The file name cannot contain the following characters: \/: *? "< > |
title = Title.replace (' \ \ ', '). Replace ('/', '). Replace (': ', '). Replace (' * ', '). Replace ('? ', '). Replace (' > ', '). Replace (' < ', '). Replace (' | ', ')
return title
# used to store the content published by the landlord
def save_data (self,url,title,endpage):
# load the page data into the array
Self.get_data (Url,endpage)
# Open Local File
f = open (title+ '. txt ', ' w+ ')
F.writelines (Self.datas)
F.close ()
Print U ' crawler report: Files have been downloaded locally and packaged as txt files '
Print U ' Press any key to exit ... '
Raw_input ();
# get the page source code and store it in an array
def get_data (self,url,endpage):
url = url + ' &pn= '
For I in Range (1,endpage+1):
Print U ' Crawler report: Crawler%d loading ... '% i
mypage = urllib2.urlopen (url + str (i)). Read ()
# The HTML code in MyPage is processed and stored in the Datas
Self.deal_data (Mypage.decode (' GBK '))
# keying content out of the page code
def deal_data (self,mypage):
MyItems = Re.findall (' id= ' post_content.*?> (. *?) ', Mypage,re. S
For item in myitems:
data = Self.myTool.Replace_Char (Item.replace ("\ n", ""). Encode (' GBK '))
Self.datas.append (data+ ' \ n ')
#--------Program Entrance------------------
Print U "" #---------------------------------------
# program: Baidu Paste Stick Crawler
# version: 0.5
# Author: why
# Date: 2013-05-16
# language: Python 2.7
# action: Enter the URL to automatically see only the landlord and save to the local file
# function: Store the contents of the landlord post to the local TXT storage.
#---------------------------------------
"""
# Take a novel as an example
# bdurl = ' Http://tieba.baidu.com/p/2296712428?see_lz=1&pn=1 '
Print U ' Please enter the last digit string of the address: '
Bdurl = ' http://tieba.baidu.com/p/' + str (raw_input (U ' http://tieba.baidu.com/p/'))
#调用
Myspider = Baidu_spider (Bdurl)
Myspider.baidu_tieba ()
The above is the improvement after the crawl Baidu Bar all the code, very simple and practical, I hope to be helpful to everyone
