Python implements batch file download

Python implements batch file download

I have shared with you the python multi-thread web page capture function. However, this function can only capture the source code of the web page using python. If you want to download a file using python, the above may not be suitable for you, recently I encountered this problem when I used python for file download, but it was finally solved. I sent the code out.

Python implements batch file download

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

#! /Usr/bin/env python #-*-Coding: UTF-8 -*-   From gevent import monkey Monkey. patch_all () From gevent. pool import Pool Import requests Import sys Import OS   Def download (url ): Chrome = 'mozilla/5.0 (X11; Linux i86_64) AppleWebKit/123456' + '(KHTML, like Gecko) Chrome/41.0.2272.101 Safari/123' Headers = {'user-agent': chrome} Filename = url. split ('/') [-1]. strip () R = requests. get (url. strip (), headers = headers, stream = True) With open (filename, 'wb ') as f: For chunk in r. iter_content (chunk_size = 1024 ): If chunk: F. write (chunk) F. flush () Print filename, "is OK"   Def removeLine (key, filename ): OS. system ('sed-I/% s/d % s' % (key, filename ))   If _ name _ = "_ main __": If len (sys. argv) = 2: Filename = sys. argv [1] F = open (filename, "r ") P = Pool (4) For line in f. readlines (): If line: P. spawn (download, line. strip ()) Key = line. split ('/') [-1]. strip () RemoveLine (key, filename) F. close () P. join () Else: Print 'usage: python % s urls.txt '% sys. argv [0]

Methods for other netizens:

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

From OS. path import basename From urlparse import urlsplit Def url2name (url ): Return basename (urlsplit (url) [2])   Def download (url, localFileName = None ): LocalName = url2name (url) Req = urllib2.Request (url) R = urllib2.urlopen (req) If r.info (). has_key ('content-disposition '): # If the response has Content-Disposition, we take file name from it LocalName = r.info () ['content-disposition']. split ('filename= ') [1] If localName [0] = '"' or localName [0] = "'": LocalName = localName [1:-1] Elif r. url! = Url: # If we were redirected, the real file name we take from the final URL LocalName = url2name (r. url) If localFileName: # We can force to save the file as specified name LocalName = localFileName F = open (localName, 'wb ') F. write (r. read ()) F. close ()   Download (r 'url of the python file you want to download ')

The above is all the content shared in this article. You can test which method is more efficient.