Python Crawler Learning 2

This is a small white study note .... The great God do not spray.

This learning embarrassing Encyclopedia of the web crawler.

http://blog.csdn.net/pleasecallmewhy/article/details/8932310

Because that embarrassing thing Wikipedia page revision, content class has no title ... So the source code has also changed.

So find an improved version of

http://blog.csdn.net/u011350541/article/details/52264073

Thanks to the above authors for their selfless sharing.

Improve the source code:

#-*-coding:utf-8-*-import urllib2 import urllib import re import thread import time              Import JSON #-----------load handling embarrassing encyclopedia-----------class Spider_model:def __init__ (self): Self.page = 1 self.pages = [] self.enable = False # All the jokes are deducted, added to the list and returned List def getpage (self,page): Myurl = "http://m.qiushibaike.com/hot/page/" + page User_age NT = ' mozilla/4.0 (compatible; MSIE 5.5; Windows NT) ' headers = {' User-agent ': user_agent} req = Urllib2.          Request (myurl, headers = headers) Myresponse = Urllib2.urlopen (req) mypage = Myresponse.read () # print MyPage unicodepage = Mypage.decode ("Utf-8") # Find all of the class= "content" div tags #re. S is any matching pattern, that is. Can match line break myitems = Re.findall (' <div.*?class= ' content > (. *?) </div> ', Unicodepage,re. S) Items = [] # Print MyItems # print str (myitems). Decode (' String_escape ') # print json.dumps (myitems, en     Coding= "UTF-8", Ensure_ascii=false) # for item in myitems: # # The first of the item is the title of the Div, which is the time # # The second of the item is the content of the Div, which is the content # items.append ([Item[0].replace ("\ n", ""), Item[1].replace ("\ n", "")]) # Prin T myitems # Print STR (myitems). Decode (' String_escape ') # Print str (myitems). Encode ("UTF-8") # p              Rint Myitems[0] Return myitems # used to load the new satin def LoadPage (self): # runs if the user does not enter quit While self.enable: # If the contents of the pages array are less than 2 # print Len (self.pages) If Len (Self.pages) < 2:try: # get new pages in the satin mypage = SELF.G                  Etpage (str (self.page)) Self.page + = 1 self.pages.append (mypage)            Except              print ' Can't link embarrassing encyclopedia!     ' Else:time.sleep (5) # def showpage (self,nowpage,page): #          Print U '%d page '% page,json.dumps (nowpage, encoding= "UTF-8", Ensure_ascii=false) def showpage (self,nowpage,page):                   i = 0 # print len (nowpage) for I in Range (0,len (nowpage)): If I < Len (nowpage):                   Print U ' page%d, section%d stories '% (page,i), nowpage[i].replace ("\ n", "") i + = 1 Else:          Break def Start (self): self.enable = True page = self.page Print U ' is loading please wait ... ' # Create a new thread in the background to load the satin and store the Thread.start_new_thread (self. LoadPage, ()) #-----------load handle embarrassing encyclopedia-----------while self.enable: # If the Self's page array is stored                  There are elements if Self.pages:nowPage = Self.pages[0] del self.pages[0] SeLf. ShowPage (nowpage,page) page + = 1 #-----------Program Entry-----------Print U ""- --------------------------------------Program: Embarrassing reptile version: 0.3 Why Date: 2014-06-03 language: Python 2.7 operation: Input Q Uit quit reading embarrassing encyclopedia function: Press ENTER to browse today's embarrassing hot---------------------------------------"" "" Print U "press ENTER to view today's embarrassing content: ' Raw_input (') MyModel = Spider_model () Mymodel.start ()

　　

This source code can run, but I run a bit of a problem.

will appear garbled.

Then learn and try to find out why.

////////////////////////////////////////////////////////////////////////////////////////////

#重新跑了几次后

Again run will not appear garbled .... That's weird.

But there was.

Unicodeencodeerror: ' GBK ' codec can ' t encode character U ' \u22ef ' in position 13:illegal multibyte sequence
Unhandled exception in thread started by
Sys.excepthook is missing
Lost Sys.stderr

For this kind of mistake, looked up and found

http://www.crifan.com/unicodeencodeerror_gbk_codec_can_not_encode_character_in_position_illegal_multibyte_sequence/

This post is very detailed.

PS: Coding Knowledge

And then put on a computer code of the knowledge of the Post ~

https://www.zhihu.com/question/23374078

Talk about Python-encoded posts

http://lukejin.iteye.com/blog/598303

Nutshell:

Unicode is the source code, the character set digitization;

UTF8 is channel encoding for better storage and transmission.

////////////////////////////////////////////////////////////////////////////////////////////

And ran a few times, again garbled ....

I do the debugging step-by-step, but it can be run normally, each page can be displayed.

But run up is normal display a few pages, then garbled.

Don't know why ....

///////////////////////////////////////////////////////////////////////////////////////////

1. Regular expressions

Do not start learning, one-time reading will be a bit messy.

Just say the inside of this source.

Because to find the encyclopedia inside the jokes, so look at the HTML source code page. That is true.

So we're looking for <div class = "Content" >....</div> something in the middle.

Need to use Re.findall ()

Usage: Re.findall (pattern, string[, flags]): Returns the list.

Give me a chestnut, eg:

Relink for search criteria

Info for search Target

Then this time the source code is as follows:

MyItems = Re.findall (' <div.*?class= ' content > (. *?) </div> ', Unicodepage,re. S)

The meaning is:

In unicodepage with ' <div.*?class= ' content > (. *?) </div> ' This regular expression to search everything

Match our conditions <div class = "Content" >....</div>, and put the search results into the myitems list.

2.print problem

Mainly for this

print u'% d page, section%d stories '% (page,i), nowpage[i].replace ("\ n", "")

%d is a placeholder, which is to occupy a position first, then fill it back.

The padding is in the back (),%d corresponds to page, and the second%d corresponds to I.

Then output nowpage[i].replace ("\ n", "")

Replace () replaces the two newline characters in nowpage[i] "\ n" with nothing "".

The reason for this is that each line can display a piece of a piece.

/////////////////////////////////////////////////////////////////////////////////////

After commissioning

Get a general idea of how the whole source works ...

There is a thread loadpage () in the background that loads the Web page, and if Len (self.pages) is greater than the second to determine whether to load the stored Web page, load only one page at a time, which is 20 pieces.

The main process is to display the role of the Web page, put the first loaded page into the list nowpage, and then delete self.pages, so that its length len (self.pages) back to 1.

The subsequent thread can then load the second time, and then the CMD displays the first loaded Web page.

Python Crawler Learning 2