[Record] [python] python crawler, download all gallery of a certain image website,

[Record] [python] python crawler, download all gallery of a certain image website,

　　

Casual only for learning and exchange, please indicate the source when reprint, http://www.cnblogs.com/CaDevil/p/5958770.html

 

This essay records my first python program. It crawls all the atlas of the specified image site and is still a very simple single-thread program. The next step is to rewrite it into multiple threads, although python multithreading is broken down. Add exception handling.

Recently, I have been practicing the python program, and I want to write my own crawler to train my hands. During the compilation process, you may encounter various problems, such as Chinese encoding and html-less requests. This essay aims to record the problems encountered and provide corresponding solutions. Of course, these solutions are provided by others, such as East and West, ^ _ ^.

First roughly describe the problem encountered in the process of code: casually used for learning exchange, reprint please indicate the source, http://www.cnblogs.com/CaDevil/p/5958770.html

• Html Chinese Decoding

Although beautifulsoup adopts the UTF-8 encoding method by default, the encoding method of some sites in China is UTF-8. You still need to manually set the Chinese encoding method.

In BeautifulSoup, manually set the encoding method as follows:

encoding = "gb18030"soup = BeautifulSoup(html,from_encoding=encoding)　　

　　

• List append and extend use essay only for learning exchange, reprinted please indicate the source, http://www.cnblogs.com/CaDevil/p/5958770.html

Query related information. append in list inserts a value into list, while extend inserts a list into another list;

To merge list1 into list2, use list2.extend (list1 );

If a value is inserted into the list, use list. append (vale)

 

• The value returned by beautifulsoup. You can directly find the nodes that meet the conditions.

The value returned by beautifulsoup is a syntax tree. The result is similar to the structure of the html dom [this part is somewhat retained. I am not familiar with the dom structure yet]. find the corresponding tag through find and find_all. The former returns the object of a tag, and the latter returns the list. The value of this list is multiple objects.

 

• Try to disguise urllib. urlretrieve () as a user.

Before urlretrieve is used, the http request is disguised as a user to avoid the site from responding to the download request. If this parameter is not set, wireshark can be used to check that the user-agent value of the download request is related to python. The server may ignore the download request. You can also add referer. Some sites use referer to identify leeching. If this parameter is not set, the downloaded images are all anti-leech images. Casual only for learning and exchange, please indicate the source when reprint, http://www.cnblogs.com/CaDevil/p/5958770.html

# Disguise the downloaded http request. Otherwise, some sites do not respond to the download request. # If this parameter is not set, the user-agent in the download request is python + version urllib. URLopener. version = 'mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 SE 2.X MetaSr 100' # download the image to the specified directory, save the file name urllib of the image on the server. urlretrieve (imgurl, downloadpath)

　　

There is a lot of nonsense, on the code. To avoid the suspicion of promoting the image site, do not place the link of the site. The code is only used for learning and communication.

Import urllib2import ioimport randomimport urllibfrom bs4 import into reimport osimport sysreload (sys) sys. setdefaultencoding ('utf8') def getHtml (url): # Try to make crawlers appear as a normal user. If this parameter is not set, the user-agent is displayed as Python + user_agent = ['mozilla/5.0 (Windows NT 5.2) AppleWebKit/534.30 (KHTML, like Gecko) in the sent request) chrome/12.0.742.122 Safari/534.30 ', 'mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0', 'mozilla/4.0 (compatible; MSIE 8.0; windows NT 5.2; Trident/4.0 ;. net clr 1.1.4322 ;. net clr 2.0.50727 ;. NET4.0E ;. net clr 3.0.20.6.2152 ;. net clr 3.5.30729 ;. NET4.0C) ', 'Opera/9. 80 (Windows NT 5.1; U; zh-cn) Presto/2.9.168 Version/100', 'mozilla/11.50 (Windows; U; Windows NT 5.0; zh-CN) appleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1 ', 'mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0 ;. net clr 2.0.50727 ;. net clr 3.0.04506.648 ;. net clr 3.5.21022 ;. NET4.0E ;. net clr 3.0.20.6.2152 ;. net clr 3.5.30729 ;. NET4.0C) '] # sets the webpage encoding format and decodes the obtained Chinese Characters Encoding = "gb18030" # construct an http request header and Set user-agent header = {"User-Agent": random. choice (user_agent)} # construct the send request = urllib2.Request (url, headers = header) # send the request and obtain the html page html = urllib2.urlopen (request) returned by the server ). read () # Use beautifulSoup to process html pages, similar to dom soup = BeautifulSoup (html, from_encoding = encoding) return soup # Get the page number def getPageNum (url) of all gallery pages of the entire site ): soup = getHtml (url) # obtain the total page number of all the Atlas directly on the homepage of the site. nums = soup. find_all ('A', class _ = 'page-numbers ') # Remove the next page and obtain the totlePage = int (nums [-2] on the last page. text) return totlePage # get the name of the specified page set and the link def getPicNameandLink (url): soup = getHtml (url) meun = [] # similar to html dom object, directly find the ul tag whose id is "pins" and the returned result is a dom object targetul = soup. find ("ul", id = "pins") if targetul: # obtain all hyperlinks under this ul. The returned value type is list, the second parameter in find_all indicates the attribute pic_list = targetul of a specified tag. find_all ("a", target = "_ blank") if pic_list: # traverse all specified tags a for pic In pic_list: # obtain the link of the gallery. link = pic ["href"] picturename = "" # Find the img tag whose "class" is "lazy" in tag. # In "find", the second parameter indicates the attribute of a specified tag. # In python, class is a reserved word. The class attribute of all labels is named "class _" img = pic. find ("img", class _ = 'Lazy ') if img: # ensure that Chinese characters can be transcoded normally. Picturename = unicode (str (img ["alt"]) else: continue # insert gallery name and corresponding url meun. append ([picturename, link]) return meun return None # function gets all gallery names def getallAltls (url): totalpage = getPageNum (url) # Gets all gallery names on the homepage. The home page url is different from other pages. No page meun = getPicNameandLink (url) # traverse all gallery pages cyclically to obtain the gallery name and link for pos in range (2, totalpage ): currenturl = url + "/page/" + str (pos) # The value returned by getPicNameandLink () is a list. # Extend is used when a list is inserted to another list. # If you insert a value, you can use append meun. extend (getPicNameandLink (currenturl) return meun # Get all gallery names and links from the home page to the specified page def getparAltls (url, page): meun = getPicNameandLink (url) for pos in range (2, page): currenturl = url + "/page/" + str (pos) meun. extend (getPicNameandLink (currenturl) return meun # retrieve the image page number in a single album def getSinglePicNum (url): soup = getHtml (url) # pagenavi is still an object (Tag ), you can use find_all to find the specified tag. pagenavi = soup. find ("div ", Class _ = "pagenavi") pagelink = pagenavi. find_all ("a") num = int (pagelink [-2]. text) return num # download all images in a single album def getSinglePic (url, path): totalPageNum = getSinglePicNum (url) # starting from the first page, download all images in a single gallery # range () The second parameter is the upper limit of the range value. This value is not included in the loop # You need to add 1 to ensure that all pages are read. For I in range (1, totalPageNum + 1): currenturl = url + "/" + str (I) downloadpic (currenturl, path) # download the image def downloadpic (url, path) on a single page: soup = getHtml (url) # Find out the parent container div pageimg = soup where the specified image is located. find ("div", class _ = "main-image") if pageimg: # find the img In the div container, which contains only one img = pageimg. find ("img") # Get the image url imgurl = img ["src"] # Get the image file name restring = R' [A-Za-z0-9] + \. jpg 'reimgname = re. findall (restring, imgurl) # convert the image Saved in the specified directory path = str (path) if path. strip () = "": downloadpath = reimgname [0] else: downloadpath = path + "/" + reimgname [0] # disguise the downloaded http request, otherwise, some sites do not respond to the download request. # If this parameter is not set, the user-agent in the download request is python + version urllib. URLopener. version = 'mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 SE 2.X MetaSr 100' # download the image to the specified directory, save the file name urllib of the image on the server. urlretrieve (imgurl, downloadpath) def downimgofsite (url, path = ""): path = str (path) # Get the name and link of all the Atlas meun_list = getallAltls (url) directorypath = "" for meun in meun_list: directoryname = Meun [0] if path. strip ()! = "": Directorypath = path + "/" + directoryname else: directorypath = OS. getcwd + "/" + directoryname if not OS. path. exists (directorypath): OS. makedirs (directorypath) getSinglePic (meun [1], directorypath) if _ name _ = "_ main __": # page = 8 url = "XXXXX" menu = getallAltls (url) # menu = getparAltls (url, page) f = open ("tsts.txt", "") for I in menu: f. write (str (unicode (I [0]) + "\ t" + str (I [1]) + "\ n") f. close ()View Code