Use python to crawl the photos in the douyu app,

Use python to crawl the photos in the douyu app,

Preface

I didn't expect python to be so powerful and fascinating. I used to see a picture as a copy and paste. Now, I can use a program to save a picture after learning python.

I recently saw that the pictures in douyu were good. I decided to use the latest python Technology for crawling. I will share the implementation process for your reference. I will not talk much about it below, let's take a look at the detailed introduction.

The method is as follows:

First download a douyu (you can do it without downloading it. The URLs are all here, right)

Capture a json data packet and get the following address.

 

According to the test results, modifying the offset value is equivalent to page flip of the app.

Access this url and return a large dictionary with two indexes, one error and one data. Data is an array of 20 characters, and each array is a dictionary. Each dictionary has another index, vertical_src.

Our goal is it!

import urllib.parseimport urllibimport jsonimport urllib.requestdata_info={}data_info['type']='AUTO'data_info['doctype']='json'data_info['xmlVersion']='1.6'data_info['ue']='UTF-8'data_info['typoResult']='true'head_info={}head_info['User-Agent']='DYZB/2.271 (iphone; iOS 9.3.2; Scale/3.00)'url='http://capi.douyucdn.cn/api/v1/getVerticalRoom?aid=ios&client_sys=ios&limit=20&offset=20'data_info=urllib.parse.urlencode(data_info).encode('utf-8')print(data_info)requ=urllib.request.Request(url,data_info)requ.add_header('Referer','http://capi.douyucdn.cn')requ.add_header('User-Agent','DYZB/2.271 (iphone; iOS 9.3.2; Scale/3.00)')response=urllib.request.urlopen(requ)print(response)html=response.read().decode('utf-8')

In this case, over 20 lines of code can return json data. Then, the json code is sliced to obtain the url of each broadcaster's photo.

Then you can get the photo on this page.

import jsonimport urllib.requestdata_info={}data_info['type']='AUTO'data_info['doctype']='json'data_info['xmlVersion']='1.6'data_info['ue']='UTF-8'data_info['typoResult']='true'url+str(i)='http://capi.douyucdn.cn/api/v1/getVerticalRoom?aid=ios&client_sys=ios&limit=20&offset='+str(x)data_info=urllib.parse.urlencode(data_info).encode('utf-8')print(data_info)requ=urllib.request.Request(url,data_info)requ.add_header('Referer','http://capi.douyucdn.cn')requ.add_header('User-Agent','DYZB/2.271 (iphone; iOS 9.3.2; Scale/3.00)')response=urllib.request.urlopen(requ)print(response)html=response.read().decode('utf-8')''' print(type(dictionary))print(type(dictionary[data]))'''dictionary=json.loads(html)data_arr=dictionary["data"]for i in range(0,19):  name=data_arr[i]["nickname"]  img_url=data_arr[i]["vertical_src"]  print(type(img_url))  respon_tem=urllib.request.urlopen(img_url)  anchor_img=respon_tem.read()  with open('../photos/'+name+'.jpg','wb') as f:    f.write(anchor_img)

Then, modify the settings to enable the page flip function.

import urllib.parseimport urllibimport jsonimport urllib.requestdata_info={}data_info['type']='AUTO'data_info['doctype']='json'data_info['xmlVersion']='1.6'data_info['ue']='UTF-8'data_info['typoResult']='true'data_info=urllib.parse.urlencode(data_info).encode('utf-8')for x in range(0,195):  url='http://capi.douyucdn.cn/api/v1/getVerticalRoom?aid=ios&client_sys=ios&limit=20&offset='+str(x)  print(data_info)  requ=urllib.request.Request(url,data_info)  requ.add_header('Referer','http://capi.douyucdn.cn')  requ.add_header('User-Agent','DYZB/2.271 (iphone; iOS 9.3.2; Scale/3.00)')  response=urllib.request.urlopen(requ)  print(response)  html=response.read().decode('utf-8')  dictionary=json.loads(html)  data_arr=dictionary["data"]  for i in range(0,19):    name=data_arr[i]["nickname"]    img_url=data_arr[i]["vertical_src"]    print(type(img_url))    respon_tem=urllib.request.urlopen(img_url)    anchor_img=respon_tem.read()    with open('../photos/'+name+'.jpg','wb') as f:      f.write(anchor_img)

Then wait ~~

Summary

The above is all the content of this article. I hope the content of this article has some reference and learning value for everyone's learning or work. If you have any questions, please leave a message to us, thank you for your support.