This article mainly introduces the method for capturing NetEase News through Python regular expressions, and analyzes in detail the implementation skills and precautions related to Netease News capture operations using regular expressions in Python based on the instance form, for more information about how to capture NetEase News through Python regular expressions, combined with the instance form, this article analyzes in detail the implementation skills and precautions for using regular expressions in Python for NetEase News capture operations. For more information, see
This article describes how to capture NetEase News through Python regular expressions. We will share this with you for your reference. The details are as follows:
I wrote some crawlers about crawling NetEase News, and found that the source code of the webpage and the comments of the webpage are basically not correct. therefore, the packet capture tool is used to obtain the hidden address of the comment (each browser has its own packet capture tool, which can be used to analyze the website)
If you observe it carefully, you will find that there is a special one, so this is what you want.
The next step is the code (also according to the modification and writing of the great god ).
# Coding = utf-8import urllib2import reimport jsonimport timeclass WY (): def _ init _ (self): self. headers = {'user-Agent': 'mozilla/5.0 (Windows NT 5.1) AppleWebKit/534.24 (KHTML, like '} self. url = 'http: // Your def getpage (self, page): full_url = 'http: // your (page?}'.html 'return full_url def gethtml (self, page): try: req = urllib2.Request (page, None, self. headers) response = urllib2.urlopen (req) html = response. read () return html failed t urllib2.URLError, e: if hasattr (e, 'reason '): print u "connection failed", e. reason return None # processing string def Process (self, data, page): if page = 1: data = data. replace ('Var replyData = ', '') else: data = data. replace ('Var newPostList = ', '') reg1 = re. compile ("\ [") data = reg1.sub ('', data) reg2 = re. compile ('<\/a> \]') data = reg2.sub (', data) reg3 = re. compile ('
') Data = reg3.sub ('', data) return data # parse json def dealJSON (self): with open (" WY.txt "," a ") as file: file. write ('id' + '|' + 'comment' + '|' + 'step on' + '|' + 'top' + '\ n ') for I in range (1, 12): if I = 1: data = self. gethtml (self. url) data = self. process (data, I) [:-1] value = json. loads (data) file1_open('wy.txt ', 'A') for item in value ['hotposts']: try: file. write (item ['1'] ['F']. encode ('utf-8') + '|') file. write (item ['1'] ['B']. encode ('utf-8') + '|') file. write (item ['1'] ['A']. encode ('utf-8') + '|') file. write (item ['1'] ['V']. encode ('utf-8') + '\ n') failed t: continue file. close () print '-- collecting % d/12 --' % I time. sleep (5) else: page = self. getpage (I) data = self. gethtml (page) data = self. process (data, I) [:-2] # print data value = json. loads (data) # print value file1_open('wy.txt ', 'A') for item in value ['newposts']: try: file. write (item ['1'] ['F']. encode ('utf-8') + '|') file. write (item ['1'] ['B']. encode ('utf-8') + '|') file. write (item ['1'] ['A']. encode ('utf-8') + '|') file. write (item ['1'] ['V']. encode ('utf-8') + '\ n') failed t: continue file. close () print '-- collecting % d/12 --' % I time. sleep (5) if _ name _ = '_ main _': WY (). dealJSON ()
The above is the code I crawled.
The above is the details of the python code sample for capturing NetEase News. For more information, see other related articles in the first PHP community!