When there is nothing to do at ordinary times, I will watch Youku lol videos. At the beginning of each program, I will use a software program to draw a lottery from the audience who commented on the video. After reading it, I want to try it myself, only code, no interface
Steps:
1. When I flip the page for a comment, I find that the website has not changed. You can know that the page is obtained through ajax and direct crawling does not work.
2. Use httpWatch analysis to obtain the http://comments.youku.com/comments /~ Ajax/vpcommentContent.html? _ Ap = {% 22 videoid % 22: % 22XNjk5Mzc2NzE2% 22, % 22sid % 22: 663062266, % 22 page %, % 22last_modify % 22: 1397742780, % 22 showid % 22:0} & __ ai = & __ callback = displayComments, get the http://comments.youku.com/comments/~ through tool Analysis Ajax/vpcommentContent.html? _ Ap = {"videoid": "XNjk5Mzc2NzE2", "sid": 663062266, "page": 41, "last_modify": 1397742780, "showid ": 0} & __ ai = & __ callback = displayComments,
3. Here, you can get a comment after you change the page.
Import timeimport urllibimport reimport random # obtain the webpage source code def getHtml (x, url): # url = 'HTTP: // comments.youku.com/comments /~ Ajax/vpcommentContent.html? _ Ap = {"videoid": "XNjk5Mzc2NzE2", "sid": 663062266, "page": % d, "last_modify": 1397742780, "showid ": 0} & __ ai = & __ callback = displayComments '% x html = urllib. urlopen (url ). read () print url # print html return html # Get comments def getComment (y, html): # match comments reg = re. compile (R'
(.*?)
', '', Result) ''' for result in results: print re. sub ("", '', result) print '------------------ ''' print result return resultif _ name __= =' _ main __': num = raw_input ('Total number of input pages: ') start = time. clock () x = random. randint (0, int (num) # nth y = random. randint (0, 30) print '% d page' % x print '% d' % y url = 'HTTP: // comments.youku.com/comments /~ Ajax/vpcommentContent.html? _ Ap = {"videoid": "XNjk5Mzc2NzE2", "sid": 663062266, "page": % d, "last_modify": 1397742780, "showid ": 0} & __ ai = & __ callback = displayComments '% x html = getHtml (x, url) comment = getComment (y, html) print comment print 'done %. 2f seconds '% (time. clock ()-start)
Result:
No matter how I change the encoding method, I don't know how to make this Chinese output