Python crawls images on webpages (sogou images) for details,

Python crawls images on webpages (sogou images) for details,

Preface

In the past few days, I have studied crawler algorithms that have been very curious. Here are some of my experiences in the past few days. Enter the text below:

Work environment you may need:

Download Python 3.6 Official Website

Local download

Here we use sogou as the crawling object.

First, go to the sogou image http://pic.sogou.com/, and enter the wallpaper classification (of course, this is just an example of Q_Q). If you need to crawl the information of a website, you need to get a preliminary understanding of it...

After entering, this is the case. Then F12 enters the developer option. The author uses Chrome.

Right-click an image and choose check.

We found that the image src is under the img label, so we tried to extract the component with the Python requests, then obtain the src of img and then use urllib. request. urlretrieve download images one by one, so as to obtain data in batches, the idea is good, the following should tell the program to crawl the url for the http://pic.sogou.com/pics/recommend? Category = % B1 % DA % D6 % BD. This url is from the address bar after category. Now that we understand the url address, let's get started with the pleasant code:

When writing this crawler program, it is best to debug it step by step to ensure that every step of our operation is correct. This is also a good habit of programmers. I don't know if I am a programmer. We will analyze the web pages that the url points.

import requestsimport urllibfrom bs4 import BeautifulSoupres = requests.get('http://pic.sogou.com/pics/recommend?category=%B1%DA%D6%BD')soup = BeautifulSoup(res.text,'html.parser')print(soup.select('img'))

Output:

It is found that the output content does not contain the image elements we want, but only analyzes the img of the logo, which is obviously not what we want. That is to say, the image data is not in the url that is http://pic.sogou.com/pics/recommend? Category = % B1 % DA % D6 % BD. Therefore, this element may be dynamic. Careful students may find that when you move the scroll wheel down within the webpage, the image is dynamically refreshed. That is to say, this webpage does not load all resources at a time, but dynamically loads resources. This also avoids the loading speed from being affected because the webpage is too bloated. The following painful exploration begins. We are trying to find the true URLs of all images. I am also new to them. I am not very experienced in finding these URLs. Final Location: F12> Network> XHR> (click File under XHR)> Preview.

We found that it was a bit close to the elements we needed. Open all_items and find that the following elements are 0 1 2 3... one by one. Try to open a url. The image address is found to be true. Find the target. Click Headers under XHR

Obtain the second line.

Request URL:

Http://pic.sogou.com/pics/channel/getAllRecomPicByTag.jsp? Category = % E5 % A3 % 81% E7 % BA % B8 & tag = % E5 % 85% A8 % E9 % 83% A8 & start = 0 & len = 15 & width = 1536 & height = 864, try removing unnecessary parts. The trick is that the access will not be affected after the possible parts are deleted. It is filtered by the author. The final url: http://pic.sogou.com/pics/channel/getAllRecomPicByTag.jsp? Category = % E5 % A3 % 81% E7 % BA % B8 & tag = % E5 % 85% A8 % E9 % 83% A8 & start = 0 & len = 15 literal meaning, it is known that category may be followed by classification. Start is the start subscript, len is the length, that is, the number of images. Now, let's get started with the pleasant code:

The development environment is Win7 Python 3.6. When running Python, you need to install requests,

Install requests in Python3.6 and press CMD to enter:

pip install requests

I am also debugging and writing here. Here I will post the final code:

Import requestsimport jsonimport urllibdef getSogouImag (category, length, path): n = length cate = category imgs = requests. get ('HTTP: // logs? Category = '+ cate +' & tag = % E5 % 85% A8 % E9 % 83% A8 & start = 0 & len = '+ str (n) jd = json. loads (imgs. text) jd = jd ['all _ items '] imgs_url = [] for j in jd: imgs_url.append (j ['bthumburl']) m = 0 for img_url in imgs_url: print ('***** '{str(m={'.jpg ******' + 'downloading... ') urllib.request.urlretrieve(img_url,path+str(m?#'.jpg') m = m + 1 print ('Download complete! ') GetSogouImag ('wallpaper', 2000, 'd:/download/wallpaper /')

I was a little excited when the program ran. Come on, feel:

 

 

So far, the programming process of the crawler has been described. In general, finding the url where the element to be crawled is the key to crawling many links

Summary

The above is all the content of this article. I hope the content of this article will help you in your study or work. If you have any questions, please leave a message, thank you for your support.