Python network data Capture II (Bing image crawl)

The previous attempt to crawl Baidu hot data, this time continue to choose to use Bing Search to grab pictures under the practice, the code in the bottom for your reference. The program needs to pass in three parameters, picture keywords, picture save path, need to crawl the number. There may be some errors during the run (mostly network errors, such as timeouts, etc.) I capture here only to print out and then skip. The URL request to page through the code is captured (without writing the whole, there are several parameters do not know what meaning removed.) ), and then just parse the returned HTML to extract the desired data. Here is a result of my run for reference:

Results:

Source:

1 #-*-coding:utf-8-*-2 3  fromBs4ImportBeautifulSoup4 Importurllib.request5 ImportRequests6 Import Time7 ImportJSON8 ImportSYS9 ImportReTen ImportOS One  A  - #crawl destination Site URL -Crawl_target_url ='https://cn.bing.com/images/async?q=%s&first=%d&count=%d&relp=%d&lostate=r&mmasync=1 ' the  - #number of images per fetch (35 is the number of pages per page) -Nums_per_crawl = 35 -  + #Crawl picture minimum size (in bytes), less than this value discarded -Min_image_size = 50000 +  A  at defget_image (URL, Path, count): -     Try: -u = Urllib.request.urlopen (URL, timeout=5) -t =U.read () -         ifSys.getsizeof (t) <min_image_size: -             return-1 in     exceptException as E: -         Print(URL, e) to         return-2 +     #Extract Picture Format -FRMT = Url[url.rfind ('.'):] thep = re.compile ("^\\. [A-za-z]+") *m =P.match (FRMT) $FRMT =m.group (0)Panax Notoginseng     Try: -f = open (Os.path.join (path, str (count) +frmt),'WB') the f.write (t) + f.close () A     exceptException as E: the         Print(Os.path.join (Path, str (count) +frmt), E) +         return-3 -     return0 $  $  - defCrawl_data (info, path, num): -First =0 theCount =0 -     #Create a sessionWuyis =requests. Session () the      while(Count <num): -U = crawl_target_url%(info, first, nums_per_crawl, Nums_per_crawl) Wu         #3.05s is the send time-out time, 10s is the received data time-out time -req = s.get (URL =u, timeout= (3.05, 10)) AboutBF = BeautifulSoup (Req.text,"Html.parser") $Imgtags = Bf.find_all ("a", Class_ ="IUSC") -          forEinchImgtags: -             ifCount = =Num: -                 return AUrldict = Json.loads (E.get ('m')) +             ifGet_image (urldict["Murl"], path, count) <0: the                 Continue -             Print("page%d picture download complete, total progress%d%%"% (count+1, (count+1) *100/num)) $ Sys.stdout.flush () theCount =count+1 theTime.sleep (0.01) theFirst = first +Nums_per_crawl theTime.sleep (0.1) -  in  the " " the parameter 1: Search keywords About parameter 2: Picture storage path the Parameter 3: number of fetches the " " the if __name__=='__main__': +Tstart =time.time () -Crawl_data (Sys.argv[1], sys.argv[2], int (sys.argv[3])) the     Print("All pictures downloaded, total time%.2fs"% (Time.time ()-tstart))

Python network data Capture II (Bing image crawl)