Python crawler's reading Call transfer (ii)

Tag: Unicode Text Port res otherwise headers www user article

Last blog we succeeded in crawling a chapter of a novel from a Webpage. Naturally. Next we're going to climb down the whole novel. First of all. We have to end the process from the original reading of a chapter, and then we can continue to read the next chapter after reading the Chapter.

Notice that each novel Chapter page has a link to the next page below.

By viewing the Web page source Code. Sort it out a little bit (  Not shown). We can see that this part of the HTML is in the following format:

<div id= "footlink" >  <script type= "text/javascript" charset= "utf-8" src= "/scripts/style5.js" ></ script>  <a href= "http://www.quanben.com/xiaoshuo/0/910/59301.html" > previous </a>  <a href= " http://www.quanben.com/xiaoshuo/0/910/"> return folder </a>  <a href=" http://www.quanben.com/xiaoshuo/0/ 910/59303.html "> Next </a></div>

Previous page back to folder the next page is in a div with ID footlink. If you want to match each link, you crawl a lot of other links on the page, but the Footlink div just has one! We can match this div to. Grab it and then match the <a> link in the captured div, just Three. Just take the last link is the URL of the next page, use this URL to update the target URL we crawled, so that we can always catch the next Page. The user reading logic is a chapter after each reading. Wait for user Input. Let's say quit quits the program, otherwise the next chapter is Displayed.

Basic Knowledge:

The basics of the previous article plus the Python thread Module.

source:

#-*-coding:utf-8-*-import urllib2import reimport threadimport chardetclass book_spider:def __init__ (self): Self.pages = [] Self.page = 1 Self.flag = True Self.url = "HTTP://WWW.QUANBEN.COM/XIAOSHUO/10/10412/2 095096.html "# will crawl a section def getpage (self): myurl = self.url user_agent = ' mozilla/4.0 (compatible; MSIE 5.5; Windows NT) ' headers = {' user-agent ': user_agent} req = urllib2. Request (myurl, headers = Headers) Myresponse = urllib2.urlopen (req) mypage = Myresponse.read () charse            t = chardet.detect (mypage) charset = charset[' encoding '] if charset = = ' Utf-8 ' or charset = = ' UTF-8 ': MyPage = MyPage Else:mypage = mypage.decode (' gb2312 ', ' ignore '). encode (' utf-8 ') unicodepage = Mypage.decode ("utf-8") # Find the id= "content" of the div tag try: #抓取标题 my_title = Re.search (' < H1> (. *?) 
' Return False try: #抓取章节内容 my_content = re.search (' <div.*?id= ' Htmlcont ENT "class=" contentbox ">" (. *?) <div ', Unicodepage,re. S) my_content = My_content.group (1) except:print "content HTML changes, please analyze Again. "return False my_content = my_content.replace (" <br/> "," \ n ") my_content = My_conte            Nt.replace ("", "") #用字典存储一章的标题和内容 onepage = {' title ': my_title, ' content ': my_content} try: #找到页面下方的连接区域 Foot_link = Re.search (' <div.*?
class= "chapter_turnpage" > (. *?) </div> ', Unicodepage,re. S) Foot_link = Foot_link.group (1) #在连接的区域找下一页的连接. The third Nexturl = Re.findall (u ' <a.*?href= "(. *)") according to the characteristics of the Web Page. *?> (. *?) </a> ', Foot_link,re. S) nexturl = nexturl[2][0] # Update Next crawl link Self.url = Nexturl except:pr            int "bottom link change, please analyze again!" return False return Onepage # for loading Chapter DEF LoadPage (self): while Self.flag:if (len (self.pages )-self.page < 3): try: # get new page MyPage = self. GetPage () If MyPage = = False:print ' Crawl failed!                     ' Self.flag = False self.pages.append (mypage) except: print ' cannot connect to the Web Page.            ' Self.flag = False #显示一章 def showpage (self,curpage): print curpage[' title '] Print curpage[' ContenT '] print "\ n" user_input = raw_input ("current is chapter%d, enter read the next chapter or enter quit:"% Self.page) if (us Er_input = = ' quit '): self.flag = False print "\ n" def start (self): print u ' start reading ... N ' # Creates a new thread Thread.start_new_thread (self.                LoadPage, ()) # assumes that the Self's page array contains elements while Self.flag:if self.page <= Len (self.pages): Nowpage = self.pages[self.page-1] self. ShowPage (nowpage) Self.page + = 1 Print u "end of this reading" #-----------the entrance to the program-----------print u ""----------- ----------------------------program: Read Call transfer version number: 0.2 angryrookie date: 2014-07-07 language: Python 2.7 features: Press ENTER to navigate to the next section---------- -----------------------------"" "print u" Press enter: ' raw_input (') myBook = Book_spider () mybook.start ()

Python crawler's reading Call transfer (ii)