Step by step analysis of Baidu Music playing address, using Python crawler bulk download

Baidu Music does not need to login also can download? Is it very exciting to hear the news?

Next we open Baidu Music, casually open a song, switch to the Baidu play page:

I use the Firfox browser, open Firebug first clear all the requests,

Now we re-refresh the next page, see this. mp3 address is the address of Baidu Music, we can directly copy to the Thunder download, but this approach is too elementary, if there are many songs, each copy, it is very troublesome ah. , and then we continue to analyze.

This link has a feature, that is music/1658513 what is this? You guessed right, this is the ID of each song, and then look at the following parameters Xcode this is a GUID, after comparison, each is different, this from where AH. We continue to analyze other requests:

Haha, or we find it, this songlink is the address of the music, but this request is how to come, we continue to look up, we see the POST request parameters, there is a songids this is the ID of each song,

At this point, everything went well and the address was copied, and then our Python appeared.

The environment used here is Python3.4, third-party library Beautifulsoup,requests, how to install, online has a lot, next on the code

Create a new XML file named Music.xml in the following format

<?XML version= "1.0"encoding= "Utf-8"?><Root> <URL>http://music.baidu.com/tag/Pure </URL> <pageSize>40</pageSize> <Saveplay>d:\\ Pure \\</Saveplay></Root>

URL is the category of Baidu Music address PageSize is the number of pages to download, Saveplay is the path to save

Next we build a PY file main code section:

Simulate browser requests to prevent blocking

headers={' User-agent ':' mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) gecko/20100101 firefox/44.0 ',' Referer ':' http://play.xml.baidu.com/',' Accept ':' text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 ',' accept-encoding ':' gzip, deflate, BR ',' Accept-language ':' zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 ',' Cache-control ':' max-age=0 ',' Connection ':' keep-alive '        } Methods for reading XML files

Gettagtext (TAG):        "Dom=xml.dom.minidom.parse (" Play.xml ")    node = Dom.getelementsbytagname (tag) [0]    node.childnodes:          (node. Text_node, NODE. Cdata_section_node):            rc=node.data    RC

Parsing HTML methods

defdownhtml (URL):Try: Saveplay=gettagtext ("Saveplay")Print("Ready to start Parsing page:"+url+"Please wait ...") Html=requests.get (URL,Headers=headers,Timeout= -) html.encoding="Utf-8"           Soup=beautifulsoup (Html.text,"Html.parser") Div_html=soup.find ("Div",Class_="Search-song-list song-list song-list-hook") Span_html=re.findall (' <span class= ' Music-icon-hook ' data-musicicon=\ '(.*?)\ '> ',Str(div_html)) forVinchSpan_html:data=json.loads (v) play=data["id"],data["SongTitle"] Queue.append (play) whileQueue:time.sleep (5) Music_tuple=queue.popleft () Playurl=music.replace ("$0$", music_tuple[0])Print(music_tuple[1]+". mp3 Enter the download channel and start waiting in line ...\ n") Resultjson=requests.get (Playurl,Headers=headers,Timeout= -) Data=resultjson.json ()if notdata[' Data ']:PassElse: v=data["Data"]["SongList"][0];Print("Downloading,"+v["Songname"]+". mp3 ...\ n") Time.sleep (2) Request.urlretrieve (v["Songlink"],saveplay+v["Songname"]+". mp3")Print(v["Songname"]+". mp3 download complete, download path:"+saveplay+v["Songname"]+". mp3")except:Pass

if__name__=="__main__": start = Time.time () pagesize=int(Gettagtext ("PageSize")) pageindex= -    Url=gettagtext ("url") playurls.append (URL)ifPagesize>1: forVinchRange(pageSize):ifV>0: purl=url+"? start="+Str(PageIndex) +"&size=25&third_type=0"                Playurls.append (purl) pageindex+= -     withPool (4) asP:p.map (Downhtml,playurls)Print("This download is shared:"+time.time ()-start)
Finally, let's take a look at the effect of our operation,

Step by step analysis of Baidu Music playing address, using Python crawler bulk download