Share a way for Python to crawl popular comments on NetEase cloud Music

This article will detail the examples of Python's popular comments on getting NetEase cloud music. Have a good reference value, follow the small series together to see it

Recently in the study of text mining related content, so-called paddle, in order to carry out text analysis, first get a text bar. There are many ways to get text, such as downloading out-of-the-box text documents from the Web, or getting data from a third-party API. But sometimes the data we want is not directly available because we do not provide a direct download channel or API for us to get the data. So what should we do at this time? There is a better way to do this is through a web crawler, that is, to write computer programs disguised as users to obtain the desired data. With the efficient use of computers, we can get data quickly and easily.

So how do you write a reptile? There are many languages that can write reptiles, such as Java,php,python, I personally prefer to use Python. Because Python not only has built-in powerful network library, there are many excellent third-party libraries, others have directly built the wheel, we can directly take over the use of it, which has brought great convenience to write crawler. It is not an exaggeration to say that using less than 10 lines of Python code can actually write a small crawler, while the use of other languages can write a lot more code, simple and easy to understand is the great advantage of Python.

Okay, nonsense, don't say much, get to the point of today. In recent years, NetEase Cloud music fire up, I myself is NetEase cloud music user, spent a few years. Used to be QQ music and cool dog, through my own personal experience, I think NetEase cloud music is the best feature of its accurate song recommendations and unique user reviews (solemnly declare!!!) This is not soft text, non-advertising!!! On behalf of the individual views, not happy to spray! ）。 Often a song below will be some of the many divine reviews that have been praised. Plus a few days ago NetEase cloud Music will select user reviews moved on the subway, NetEase cloud music commentary and fire again. So I want to analyze the comments of NetEase Cloud, find out the laws, especially the analysis of some of the common characteristics of thermal evaluation. With this purpose, I started to the NetEase Cloud review of the crawl work.

Python has built-in two network libraries Urllib and URLLIB2, but these two libraries are not particularly convenient to use, so here we use a highly acclaimed third-party library requests. Using requests only a few lines of code can be implemented to set up proxies, simulation of landing and other more complex crawler work. If you have already installed Pip, you can install it directly using PIP install requests. Chinese document address In this http://docs.python-requests.org/zh_CN/latest/user/quickstart.html, you have any questions you can refer to the official documents, the above will be very detailed introduction. As for the Urllib and urllib2 these two libraries is also more useful, if there is a chance I will give you another introduction.

Before the beginning of the introduction of reptiles, first of all, the basic work of the crawler, we know that we open a browser to access a URL is essentially a request to the server, the server after receiving our request, will be based on our request to return data, and then through the browser to parse the data well, Present in front of us. If we use the code, we should skip this step of the browser, send a certain amount of data directly to the server, and then retrieve the data returned by the server and extract the information we want. The problem is that sometimes the server needs to validate the request we send, and if it thinks our request is illegal, it will not return the data or return the wrong data. So in order to avoid this situation, we sometimes need to disguise the program as a normal user, in order to get a smooth response to the server. How to disguise it? This depends on the difference between a Web page accessed by a user through a browser and a Web page that we access through a program. In general, we access a Web page through a browser, in addition to sending access to the URL, but also to send additional information to the service, such as headers (header information), which is equivalent to the request for proof of identity, the server saw this data, we will know that we are through the normal browser access, We'll be back to the data. So our program has to be like a browser, when sending a request, bring the information that marks our identity so that we can get the data smoothly. Sometimes we have to log in to get some data, so we have to impersonate the login. In essence, through the browser login is post some form information to the server (including user name, password and other information), the server check after we can successfully login, the use of the same program, the browser post what data, we send it as is. As for the demo login, I'll give you a special introduction later. Of course things will not be so smooth sometimes, because some sites have set up anti-crawling measures, such as if the access is too fast, sometimes will be blocked IP (typical such as watercress). This time we have to set up a proxy server, that is, change our IP address, if an IP is blocked, then another IP, specifically how to do, these topics slowly after.

Finally, let me introduce a little trick that I think is very useful in the process of writing reptiles. If you're using Firefox or Chrome, you might notice a place called developer tools (Chrome) or a Web console (Firefox). This tool is very useful, because using it, we can clearly see in the process of accessing a website, what information the browser sends, what information the server actually returns, this information is the key to our crawler writing. Below you will see the great usefulness of it.

----------------------------------------------------the beginning of the split line---------------------------------------------------

First open the web version of NetEase Cloud music, choose a song to open its Web page, here I take Jay Chou's "Sunny days" for example. such as 1

Figure 1

Next Open the Web Console (Chrom Open Developer tools, if other browsers should also be similar), such as 2

Figure 2

Then this time we need to select the network, clear all the information, and then click Resend (equivalent to refresh the browser), so that we can intuitively see what information the browser sent and what information the server responded to. such as 3

Figure 3

The data obtained after the refresh is shown in 4:

Figure 4

You can see that the browser sends a lot of information, so which one is what we want? Here we can make a preliminary judgment by the status Code, status code (state codes) flag the status of the server request, where the status code of 200 means that the request is normal, and 304 is not normal (there are many types of status code, if you want to learn more about the self-search, Here does not say 304 specific meanings). So we usually just look at the status code of 200 request, and there is, we can use the preview in the right column to roughly see what information the server returned (or to see the response). As shown in 5:

Figure 5

By combining these two methods, we can quickly find the request we want to analyze. Note that the request URL column in Figure 5 is the URL we want to request, there are two ways to request: Get and post, there is a need to focus on the request header, which contains user-agent (client information), refrence (from where to jump) and other information, Generally, both the get and post methods carry the head information. The header information is shown in 6:

Figure 6

It is also important to note that a GET request typically takes the requested parameter directly. Parameter1=value1&parameter2=value2 and so on, so there is no need to bring extra request parameters, and the post request generally need to take extra parameters, and not directly put the parameters in the URL, So there are times when we need to focus on the column of parameters. After careful searching, we finally found the request that was originally related to the comment in the http://music.163.com/weapi/v1/resource/comments/R_SO_4_186016?csrf_token= request, as shown in 7:

Figure 7

To open this request, we found it to be a POST request with two parameters, one for params and one for Encseckey, and the values for these two parameters are very long and should feel like encrypted. As shown in 8:

Figure 8

The data returned by the server is in JSON format and contains very rich information (such as information about the reviewer, comment date, number of likes, comments, etc.), as shown in 9: (In fact hotcomments is a popular comment, comments is an array of comments)

Figure 9

At this point, we have determined the direction, that is, only need to determine the params and Encseckey parameter values can be, this problem bothered me one afternoon, I did not understand the two parameters of the encryption method, but I found a rule, http://music.163.com The number behind r_so_4_ in/weapi/v1/resource/comments/r_so_4_186016?csrf_token= is the ID value of the song, and for the Param and encseckey values of the different songs, If a song such as the two parameter value of a to the song, then for the same number of pages, this parameter is common, that is, the first page of a two parameter values to any other song two parameters, you can get the first page of the corresponding song comments, for the second page, the third page is similar. But unfortunately, the different page parameters are different, this approach can only crawl a limited number of pages (of course, crawl comments and popular comments are enough), if you want to crawl all the data, you must understand the two parameter values of the encryption method. Thought I did not understand, last night I took this question to know the search, incredibly really I found the answer. So far, how to crawl NetEase cloud music Comments All the data are finished.

By convention, the final code, the pro-test is effective:

#!/usr/bin/env python2.7#-*-coding:utf-8-*-# @Time: 2017/3/28 8:46# @Author: lyrichu# @Email: 919987476@qq.com# @Fi le:NetCloud_spider3.py "@Description: NetEase Cloud Music Review Crawler, you can fully crawl the entire review section reference the @ Small-breasted Fairy article (address: https://www.zhihu.com/question/ 36081767) Post Encryption section is also given, you can refer to the original posts: Author: Small chest Fairy Link: https://www.zhihu.com/question/36081767/answer/140287795 Source: Know from Crypto.cipher import aesimport base64import requestsimport jsonimport codecsimport time# Header information headers = {' Host ': ' music.16 3.com ", ' accept-language ':" zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 ", ' accept-encoding ':" gzip, deflate ", ' Content-type ' : "application/x-www-form-urlencoded", ' Cookie ': "_ntes_nnid=754361b04b121e078dee797cdb30e0fd,1486026808627; _NTES_NUID=754361B04B121E078DEE797CDB30E0FD; Jsessionid-wyyy=yfqt9ofhy%5ciynkxw71tqy5otszyje%2foswggtl4dmv3oa7%5cq50t%2fvaee%2fmsscifhe0tgtrmyhsppr20i%5cro %2bo%2b9pbbjnruvgzkibhnqw3tlgn%5coil%2frw7zfzzwsa3k9gd77mpsvh6fnv5hit8ms70mnb3cxk5r3ecj3tfmlwfbfozmgw%5c% 3a1490677541180; _iuqxldmzr_=32; Vjuids=c8ca7976.15a029d006a.0.51373751e63af8; vjlast=1486102528.1490172479.21; __GADS=ID=A9EED5E3CAE4D252:T=1486102537:S=ALNI_MB5XX2VLKJSIU5CIY91-TOUDOFXIW; vinfo_n_f_l_n3=411a2def7f75a62e.1.1.1486349441669.1486349607905.1490173828142; p_info=m15527594439@163.com|1489375076|1|study|00&99|null&null&null#hub&420100#10#0#0|155439 &1|study_client|15527594439@163.com; Ntes_cmt_user_info=84794134%7cm155****4439%7chttps%3a%2f%2fsimg.ws.126.net%2fe%2fimg5.cache.netease.com%2ftie% 2fimages%2fyun%2fphoto_default_62.png.39x39.100.jpg%7cfalse%7cbte1nti3ntk0ndm5qde2my5jb20%3d; usertrack=c+5+hljhgu0t1fdma66mag==; province=027; city=027; _ga=ga1.2.1549851014.1489469781; __utma=94650624.1549851014.1489469781.1490664577.1490672820.8; __utmc=94650624; __utmz=94650624.1490661822.6.2.utmcsr=baidu|utmccn= (organic) |utmcmd=organic; playerid=81568911; __utmb=94650624.23.10.1490672820 ", ' Connection ':" keep-alive ", ' Referer ': ' http://music.163.com/'}# set proxy server proxies= {' http: ': ' http://121.232.146.184 ', ' https: ': ' https:/The value of/144.255.48.197 '}# offset is: (Comments page-1) *20,total The first page is true, the remaining pages are false# First_param = ' {rid: ', offset: ' 0 ', total: ' true ', Limit: "Csrf_token", "} ' # the first parameter Second_param =" 010001 "# The second parameter # the third argument third_param =" 00e0b509f6259df8642dbc35662901477df22677ec152b5ff68ace615bb7b725152b3ab17a876aea8a5aa76d2e417629ec4ee341f56135fccf6952801 04e0312ecbda92557c93870114af6c9d05c4f7f0c3685b7a46bee255932575cce10b424d813cfe4875d3e82047b97ddef52741d546b8e289dc6935b3e Ce0462db0a22b8e7 "# fourth parameter Forth_param =" 0cojum6qyw8w8jud "# get parameter def get_params (page): # page for incoming pages IV =" 0102030405060708 "F Irst_key = Forth_param Second_key = + * ' F ' if (page = = 1): # If the first page First_param = ' {rid: ', offset: ' 0 ', total: ' true ', Lim  It: "A", Csrf_token: ""} ' H_enctext = Aes_encrypt (First_param, First_key, iv) Else:offset = str ((page-1) *20) First_param = ' {rid: ', offset: '%s ', total: '%s ', limit: ' Csrf_token ', '} '% (offset, ' false ') H_enctext = Aes_encrypt (First_param, First_key, iv) H_enctext = Aes_encrypt (H_enctext, Second_key, IV) RETurn h_enctext# get Encseckeydef get_encseckey (): Encseckey = " 257348aecb5e556c066de214e531faadd1c55d814f9be95fd06d6bff9f4c7a41f831f6394d5a3fd2e3881736d94a02ca919d952872e7d0a50ebfa1769 a7a62d512f5f1ca21aec60bc3819a9c3ffca5eca9a0dba6d6f7249b06f5965ecfff3695b54e1c28f3f624750ed39e7de08fc8493242e26dbc4484a01c 76f739e135637c "return encseckey# decryption Process def aes_encrypt (text, key, iv): Pad = 16-len (text)% Text = text + pad * CHR (PA d) encryptor = Aes.new (key, AES. MODE_CBC, iv) Encrypt_text = Encryptor.encrypt (text) Encrypt_text = Base64.b64encode (encrypt_text) return encrypt_text# Get comments JSON data def get_json (URL, params, encseckey): data = {"params": params, "Encseckey": encseckey} response = Requests.po St (URL, headers=headers, data=data,proxies = proxies) return response.content# crawl popular comments, return to heat rating list def get_hot_comments (URL): Hot_comments_list = [] hot_comments_list.append (u "User ID user nickname user avatar address Comment time likes total comments content \ n") params = Get_params (1) # first page Encseck ey = Get_encseckey () Json_text = Get_json (url,params,encsecKey) json_dict = Json.loads (json_text) hot_comments = json_dict[' hotcomments '] # Popular reviews print ("There are%d top reviews!"% Len (hot_commen TS))) for item in hot_comments:comment = item[' content '] # comment Content likedcount = item[' Likedcount '] # likes total Comment_time = Item [' Time '] # comment time (timestamp) UserID = item[' user ' [' userid '] # Reviewer ID nickname = item[' user ' [' nickname '] # nickname Avatarurl = item[' Us Er ' [' Avatarurl '] # avatar Address Comment_info = UserID + "" + Nickname + "+ Avatarurl +" "+ Comment_time +" "+ Likedcount + "" + comment + u "\ n" hot_comments_list.append (comment_info) return hot_comments_list# grab all comments on a song Def get_all_comments ( URL): All_comments_list = [] # Store all comments All_comments_list.append (u "User ID user nickname user avatar address Comment time likes total comments content \ n") # header information params = Get_p Arams (1) Encseckey = Get_encseckey () Json_text = Get_json (url,params,encseckey) json_dict = Json.loads (Json_text) comments_num = Int (json_dict[' total ')) if (comments_num% = = 0): page = comments_num/20 else:page = Int (comments_num /+ 1 Print ("Total%d comments!"% page) For I in Range (page): # page Fetch params = get_params (i+1) Encseckey = Get_encseckey () Json_text = Get_json (url,params,encseck EY) json_dict = json.loads (json_text) if i = = 0:print ("Total%d comments!"% comments_num) # Total number of comments for item in json_dict[' comments  ']: Comment = item[' content ' # comments Likedcount = item[' Likedcount '] # likes total Comment_time = item[' time '] # comment timestamp (timestamp) UserID = item[' user ' [' userId '] # Reviewer ID nickname = item[' user ' [' nickname '] # nickname Avatarurl = item[' user ' [' Avatarurl '] # Avatar address Co Mment_info = Unicode (UserID) + u "" + Nickname + U "" + Avatarurl + u "" + Unicode (Comment_time) + u "+ Unicode (Likedcou NT) + u "" + comment + u "\ n" all_comments_list.append (comment_info) Print ("Page%d crawl complete!"% (i+1)) return all_comments_list# will Comment Written to text file Def save_to_file (list,filename): With Codecs.open (filename, ' a ', encoding= ' Utf-8 ') as F:f.writelines (list) Print ("Write file succeeded!") if __name__ = = "__main__": Start_time = Time.time () # start time URL = "http://music.163.com/weapi/v1/resource/comments/R_SO_4_ 186016/?csrf_token= "filename = u" Sunny. txt "all_comments_list = get_all_comments (URL) save_to_file (all_comments_list,filename) End_time = Time.time () #结束时间 print ("The program takes%f seconds."% (End_time-start_time))

I used the above code ran a bit, caught two Jay Chou's popular songs "Sunny" (more than 1.3 million reviews) and "Advertising balloon" (more than 200,000 comments), the former ran for about more than 20 minutes, the latter has more than 6,600 seconds (that is, nearly 2 hours), as follows:

Note that I am separated by a space, each line has a user ID user nickname user avatar address Comment time likes total comments content.