Python crawler: More than 10 lines of code download King Glory all Skin __python

Access Flyai.club, one click to create your AI project

Author | Wang Qiang

SOURCE | C and Python combat

Cause: Two days ago on the public to see an article on the content is to climb the glory of the king of the skin, but the content is too much, if follow him to do certainly not come out, so I intend to do it myself.

Before the crawler or a few years ago to crawl the song of the watercress radio, then use the C++,json parsing also use a third-party library, in short, very troublesome. Recently touched Python, deep feeling that the language is really good.

Getting to the point: how to climb the heroic skin photo of the king's glory.

is divided into two steps:

Find the address of the skin picture

Download Picture 1. Find Skin Image Address

1.1 Find Heroes list

Baidu "King Glory" into the official website, into the https://pvp.qq.com/, press F12 into the debugging interface, and then press F5 Refresh interface, the logo herolist.json file is the list of heroes we need, including hero number, Hero name, Hero type, Name of the skin, etc., right-click the copy link on the file Http://pvp.qq.com/web201605/js/herolist.json

Next, check out our results:

# code fragment 1import urllib.requestimport jsonimport osresponse = Urllib.request.urlopen ("http://pvp.qq.com/web201605/js/ Herolist.json ") Hero_json = Json.loads (Response.read ()) Hero_num = Len (hero_json) print (Hero_json) print (" Hero_num: ", STR (hero_num))

The above code reads the list of heroes into the Hero_json, and gets the number of heroes, running as shown in the figure:

1.2 Find the Hero skin address

Click on the "Game Information" tab on the Home page, enter the new interface and click a hero avatar into the hero data interface, here we take Sun Shangxiang as an example:

The same F12 then F5, the mouse on the Sun Shangxiang several skin in turn, to see the debugging window

You can see that Sun Shangxiang's HD skin is 6, and we're on the first skin right button copy link get: http://game.gtimg.cn/images/yxzj/img201606/skin/hero-info/111/111- Bigskin-1.jpg, this is our dream Hero skin link.

Analyze this link, where "111" is the hero's number, and the last "1" is the hero's skin number. So far, the browser is no longer used, we have the information we have. 2. Download Pictures

2.1 heroes have several skins

In the first step to get to the Herolist.json file in the "Skin_name" field, we can only parse this field to get the number of skin and skin name. The test code (take code fragment 1) is as follows:

# code Fragment 2 Hero_name = hero_json[0][' cname '] skin_names = hero_json[0][' skin_name '].split (' | ') Skin_num = Len (skin_names) PR Int (' Hero_name: ', hero_name) print (' Skin_names: ', skin_names) print (' Skin_num: ' + str (skin_num))

The results of the operation are as follows:

You can see Lianpo total two skin, the skin name is: Just detonation and hell rock Soul.

2.2 Download Files

The download file uses the Urlretrieve interface, and the test code is as follows:

For I in Range (hero_num): # Get skin Name list skin_names = hero_json[i][' skin_name '].split (' | ') for CNT in range (len (skin_names)):  Save_file_name = ' D:\heroskin\\ ' + str (hero_json[i][' ename ']) + '-' +hero_json[i][' cname ']+ '-' +skin_names[cnt ' + '. jpg ' Skin_url = ' http://game.gtimg.cn/images/yxzj/img201606/skin/hero-info/' +str (hero_json[i][' ename ']) + '/' +str (hero_ json[i][' ename ']) + '-bigskin-' + str (cnt+1) + '. jpg ' Urllib.request.urlretrieve (skin_url, Save_file_name)

Look at the results:

So far all 224 skins have been downloaded and are all high-definition images.

It's not over yet, the program has some imperfections:

If the path D:\herolist\ does not exist, the program fails to run;

If the download fails halfway, the downloaded picture will be downloaded again when you run the program again.

Solution:

1. Check to see if the file exists, if it does not exist, create the code as follows:

# folder does not exist create Save_dir = ' D:\heroskin ' if not os.path.exists (Save_dir): Os.mkdir (Save_dir)

2. Check the existence of the file, if it exists, skip the download, the code is as follows:

If not os.path.exists (save_file_name): Urllib.request.urlretrieve (Skin_url, Save_file_name)

So far, done, paste the complete code:

#-*-Coding:utf-8-*-"" "Created on Wed Aug-23:12:17 2017@author:wangqiang" "" Import Urllib.requestimport jsonimport O Sresponse = Urllib.request.urlopen ("Http://pvp.qq.com/web201605/js/herolist.json") Hero_json = Json.loads ( Response.read ()) Hero_num = Len (hero_json) # folder does not exist then create Save_dir = ' D:\heroskin\\ ' if not os.path.exists (Save_dir): Os.mkdir (Save_dir) for I in Range (hero_num): # Get the hero skin list skin_names = hero_json[i][' skin_name '].split (' | ') for the CNT in range ( Len (skin_names)): Save_file_name = Save_dir + str (hero_json[i][' ename ']) + '-' +hero_json[i][' cname ']+ '--' +skin_names[ CNT] + '. jpg ' skin_url = ' http://game.gtimg.cn/images/yxzj/img201606/skin/hero-info/' +str (hero_json[i][' ename ']) + '/ ' +str (hero_json[i][' ename ']) + '-bigskin-' + str (cnt+1) + '. jpg ' if not os.path.exists (save_file_name): Urllib.request.urlretrieve (Skin_url, Save_file_name)

With the addition of comments and blank lines, a total of 16 lines of code achieve the function of all skin of the king's Glory, which is also excellent for the desktop background. Experience:

-end-