Simple implementation of Python crawler Images,

Simple implementation of Python crawler Images,

Python crawler image implementation

I often go to zhihu and sometimes want to save the problematic image sets. So we have this program. This is a very simple image crawler that can only crawl images that have been flushed out. Because I am not familiar with this part of content, I just want to say a few simple words and then record the code without too much explanation. If you are interested, you can use it directly. Tests are available for websites such as zhihu.

In the previous article, I shared the method of opening an image through a url. The purpose is to first check what the crawled image looks like, and then filter and save it.

Here, the requests library is used to obtain page information. It should be noted that a header is required for obtaining page information to disguise the program as a browser to access the server, otherwise, it may be rejected by the server. Then, use BeautifulSoup to filter excess information to get the image address. After obtaining the image, filter out some small pictures, such as portraits and emoticons, based on the image size. When opening or saving the image, there are more options, such as OpenCV, skimage, and PIL.

The procedure is as follows:

#-*-Coding = UTF-8-*-import requests as reqfrom bs4 import BeautifulSoupfrom PIL import Imagefrom io import BytesIOimport osfrom skimage import iourl = "https://www.zhihu.com/question/37787176" headers = {'user-agent ': 'mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Mobile Safari/100'} response = req. get (url, headers = headers) conte Nt = str (response. content) # print contentsoup = BeautifulSoup (content, 'lxml') images = soup. find_all ('img ') print u "% d images in total" % len (images) if not OS. path. exists ("images"): OS. mkdir ("images") for I in range (len (images): img = images [I] print u "processing the % d image... "% (I + 1) img_src = img. get ('src') if img_src.startswith ("http"): # use PIL '''print img_src response = req. get (img_src, headers = headers) image = I Mage. open (BytesIO (response. content) w, h = image. size print w, h img_path = "images/" + str (I + 1) + ". jpg "if w> = 500 and h> 500: # image. show () image. save (img_path) ''' # use OpenCV import numpy as np import urllib import cv2 resp = urllib. urlopen (img_src) image = np. asarray (bytearray (resp. read (), dtype = "uint8") image = cv2.imdecode (image, cv2.IMREAD _ COLOR) w, h = image. shape [: 2] print w, h img_path = "Images/" + str (I + 1) + ". jpg "if w> = 400 and h> 400: cv2.imshow (" Image ", image) cv2.waitKey (3000) # cv2.imwrite (img_path, image) # use skimage # image = io. imread (img_src) # w, h = image. shape [: 2] # print w, h # io. imshow (image) # io. show () # img_path = "images/" + str (I + 1) + ". jpg "## if w> = 500 and h> 500: ## image. show () # image. save (img_path) # io. imsave (img_path, image) print u "processed!"

Multiple options are provided for reference.