Python crawls popular songs from Netease cloud music, and python Netease

Python crawls popular songs from Netease cloud music, and python Netease

Preface

Netease cloud music is a music APP that I like and has a large number of users. The reason why many users of Netease cloud music are inseparable from its song comment function is that many Song comments are very interesting, there are also many touching comments. However, Netease cloud music does not provide hot rating rankings and comment sorting functions. It doesn't matter. This article uses crawlers to crawl those hot-rated songs on Yiyun music.

Result

For those who are not interested in the process, see here.

Ranking of songs with more than 50 thousand comments

Congratulations, Jay Chou, one of my favorite singers, has made Netease cloud music the first song to comment on millions of songs!

The results show that there are dozens of songs with over 100,000 comments:

1. Xue zhiqian is really angry now ~
2. Almost all of them are male singers. Do male singers seem to be more popular? (Don't beat me) among the male singers Jay Chou, Xue zhiqian, and Xu Ke (I like them all) Almost accounted for half of the list...
3. Fade sounds are so powerful that it is quite appealing. (It won't be able to stop writing code with Hyun Mai ..)
   

Based on the results, we made a Netease cloud music song order:

Over 100,000 songs commented

Over 50 thousand songs commented

Tip:Some of the songs with over 50 thousand comments are temporarily removed due to copyright issues and are temporarily replaced by other excellent versions.

High-energy warning: TOP 29 Lost Rivers should be played with caution. If you insist on playing, please read the comments first...

Process

1. Observe the HTML structure of Netease cloud music Official Website

Home (http://music.163.com /)

CATEGORY page (http://music.163.com/discover/playlist ).

Karaoke pages (http://music.163.com/playlist? Id = 499518394)

Song details page (http://music.163.com/song? Id = 109998)

2. Crawl the song ID

By observing the URL of the Song details page, we can find that the detailed page URL can be obtained by crawling the corresponding song ID, and the song information is on the details page. It can be seen that all the song information can be obtained as long as the ID of all the songs is collected. Where do these IDs need to be crawled? Where can I crawl from a ticket? By looking at the URL of the song list page, we can find that the song list also has an ID, and the song list ID can be crawled from the song list category page. Then, the ID of all the songs will be collected.

3. filter out matching songs by crawling the number of comments

Unfortunately, although the number of comments is also on the details page, Netease cloud music does anti-crawling and uses AJAX to call the number of comments API to fill the comment-related data, due to the asynchronous feature, the number of comments on the page we climbed to is blank, so let's look for this API. After customs clearance, observe that the XHR request is found to be the following guy ..

The response results are rich, and all the comment-related data is available. However, it is observed that this API is encrypted, but it does not matter...

4. Crawl detailed information (names, singers, etc.) of matching songs)

This step is very simple. It is easy to observe the HTML on the song details page to crawl to the name and singer information we want.

Source code

# Encoding = utf8import requestsfrom bs4 import BeautifulSoupimport OS, jsonimport base64from Crypto. cipher import AESfrom prettytable import PrettyTableimport warningswarnings. filterwarnings ("ignore") BASE_URL = 'HTTP: // music.163.com/'_session = requests. session () # match the COMMENT_COUNT_LET = 100000 class Song (object): def _ lt _ (self, other): return self. commentCount> other. commentCount # Due To Netease cloud audio Happy Song comments are filled with AJAX, so they cannot be crawled in HTML. Therefore, the comments API needs to be called, and the API is encrypted. The following is the solution def aesEncrypt (text, secKey): pad = 16-len (text) % 16 text = text + pad * chr (pad) encryptor = AES. new (secKey, 2, '20140901') ciphertext = encryptor. encrypt (text) ciphertext = base64.b64encode (ciphertext) return ciphertextdef rsaEncrypt (text, pubKey, modulus): text = text [:-1] rs = int (text. encode ('hex'), 16) ** int (pubKey, 16) % Int (modulus, 16) return format (rs, 'x '). zfill (256) def createSecretKey (size): return (''. join (map (lambda xx: (hex (ord (xx) [2:]), OS. urandom (size) [0: 16] # obtain all the song IDs of online cloud music through a third-party channel # If you are a lazy user, you can use getSongIdList () crawling from the official website is relatively time-consuming, but it is more accurate to def getSongIdListBy3Party (): pageMax = 1 # number of pages to be crawled, you can select the number of pages as needed. songIdList = [] for page in range (pageMax): Url = 'HTTP: // grri94kmi4.app.tianmaying.com/songs? Page = '+ str (page) # print url. decode ('utf-8') soup = BeautifulSoup (_ session. get (url ). content) # print soup aList = soup. findAll ('A', attrs = {'target': '_ blank'}) for a in aList: songId = a ['href ']. split ('=') [1] songIdList. append (songId) return songIdList # crawl all the songs of online music from the website> the karaoke page IDdef getSongIdList (): pageMax = 1 # number of pages to be crawled, currently, there are 42 pages in total. It takes a long time to crawl 42 pages. You can select the number of pages as needed. songIdList = [] for I in range (1, page Max + 1): url = 'HTTP: // music.163.com/discover/playlist /? Order = hot & cat = All & limit = 35 & offset = '+ str (I * 35) url. decode ('utf-8') soup = BeautifulSoup (_ session. get (url ). content) aList = soup. findAll ('A', attrs = {'class': 'tit f-thide s-fc0 '}) for a in aList: uri = a ['href '] playListUrl = BASE_URL + uri [1:] soup = BeautifulSoup (_ session. get (playListUrl ). content) ul = soup. find ('ul ', attrs = {'class': 'F-hide'}) for li in ul. findAll ('lil'): songId = (li. find ('A ')) ['Href ']. split ('=') [1] print 'successfully crawled the song ID->' + songId songIdList. append (songId) # duplicate songs are inevitable in the song list. Click the duplicate song ID songIdList = list (set (songIdList )) return songIdList # whether the number of comments matching the song meets the requirements # The number of let comments is greater than the value def matchSong (songId, let ): url = BASE_URL + 'weapi/v1/resource/comments/r_o4 _ '+ str (songId) + '/? Csrf_token = 'headers = {'cooker': 'appver = 1.5.0.75771; ', 'Referer': 'http: // music.163.com/'} text = {'username ':'', 'Password': '', 'rememberlogin': 'true'} modulus = 'invalid login' 9dc6935b3ece0462db0a22b8e7 'nonce = '0cojum6qyw8w8jud 'pubKey = '000000' text = json. dumps (text) secKey = createSecretKey (16) encText = aesEncrypt (text, nonce), secKey) cipher = rsaEncrypt (secKey, pubKey, modulus) data = {'params ': encText, 'enabledkey': encSecKey} req = requests. post (url, headers = headers, data = data) total = req. json () ['Total'] if int (total)> let: song = Song () song. Id = songId song. commentCount = total return song # Set the song information def setSongInfo (song): url = BASE_URL + 'song? Id = '+ str (song. id) url. decode ('utf-8') soup = BeautifulSoup (_ session. get (url ). content) strArr = soup. title. string. split ('-') song. singer = strArr [1] name = strArr [0]. encode ('utf-8') # Remove the words () after the song name. If you do not want to remove the words, you can note the following three lines of code index = name. find (') if index> 0: name = name [0: index] song. name = name # retrieve the qualified song list def getSongList (): print '# crawling the song number... ##' # songIdList = getSongIdList () songIdList = getSong IdListBy3Party () print '# after the song number is crawled, It is crawled to' + str (len (songIdList )) + 'beginning ## 'songlist = [] print' # Crawling a song that matches the comment count greater than '+ str (COMMENT_COUNT_LET) +... # 'for id in songIdList: song = matchSong (id, COMMENT_COUNT_LET) if None! = Song: setSongInfo (song) songList. append (song) print 'successfully matches one {Name: ', song. name, '-', song. singer, ', comment count:', song. commentCount, '} 'print' # The crawling is complete. The total number that meets the condition is' + str (len (songList) + 'beginning # 'Return songListdef main (): songList = getSongList () # sort by comment from high to low. sort () # printed result table = PrettyTable ([u'ranking', u'comment number', u'song name', u'singles']) for index, song in enumerate (songList): table. add_row ([index + 1, song. commentCount, song. name, song. singer]) print table print 'end' if _ name _ = '_ main _': main ()

Tip:With the change of Netease cloud music website structure, interface, and encryption method, this Code may not work well, but the process and principle are the same, here we will share with you the process.

Summary

The above is all about this article. I hope this article will help you in your study or work. If you have any questions, please leave a message.