This article mainly introduces the python-based simple crawling function. interested friends can refer to some nice pictures that we often see when browsing the web pages on the Internet, we want to save and download these images, or use them as desktop wallpapers or design materials.
The most common practice is to right-click and choose Save. However, when you right-click an image, it is not saved as an option. you can use a tool to extract it, but this reduces the image definition. Okay ~! In fact, you are very good. right-click to view the page source code.
We can use python to implement such a simple crawler function and crawl the code we want locally. The following describes how to use python to implement such a function.
1. obtain the data of the entire page
First, we can obtain the entire page information of the image to be downloaded.
Getjpg. py
#coding=utf-8import urllibdef getHtml(url): page = urllib.urlopen(url) html = page.read() return htmlhtml = getHtml("http://tieba.baidu.com/p/2738151262")print html
Urllib moduleProvides an interface for reading web page data. we can read www and ftp data just like reading local files. First, we define a getHtml () function:
Urllib. urlopen () methodOpen a URL.
Read () methodIt is used to read data from a URL, pass a URL to the getHtml () function, and download the entire page. The execution program prints the entire webpage.
2. filter the desired data on the page
Python provides very powerful regular expressions. we need to know a little about python regular expressions first.
If we find some beautiful wallpapers on the Baidu Post Bar, go to the front-end to view the tools. The image address is found, for example, src = "http://imgsrc.baidu.com/forum.....jpg?pic_ext="
The modification code is as follows:
import reimport urllibdef getHtml(url): page = urllib.urlopen(url) html = page.read() return htmldef getImg(html): reg = r'src="(.+?\.jpg)" pic_ext' imgre = re.compile(reg) imglist = re.findall(imgre,html) return imglist html = getHtml("http://tieba.baidu.com/p/2460150866")print getImg(html)
We have created the getImg () function to filter the image links on the entire Retrieved page. The re module mainly contains regular expressions:
- Re. compile () can compile a regular expression into a regular expression object.
- The re. findall () method reads data that contains imgre (regular expression) in html.
Run the script to obtain the URL address of the image on the page.
3. Save the filtered data on the page to the local device.
The filtered image address is traversed through a for loop and saved locally. the code is as follows:
#coding=utf-8import urllibimport redef getHtml(url): page = urllib.urlopen(url) html = page.read() return htmldef getImg(html): reg = r'src="(.+?\.jpg)" pic_ext' imgre = re.compile(reg) imglist = re.findall(imgre,html) x = 0 for imgurl in imglist: urllib.urlretrieve(imgurl,'%s.jpg' % x) x+=1html = getHtml("http://tieba.baidu.com/p/2460150866")print getImg(html)
The core here is to use the urllib. urlretrieve () method to directly download remote data to the local device.
You can use a for loop to traverse the obtained Image connections. to make the image file name look more standard, rename it, and add 1 to the naming rule using the x variable. The default storage location is the storage directory of the program.
After the program runs successfully, the downloaded file is displayed in the directory.
Result Display:
The above is all of the content in this article, hoping to help you implement the Python crawler function.