Python download File & anti-theft chain

Accidentally download a type of resources, found a lot of pages, right-click Save What, Patience.

decided to write a few lines of code in Python. The core code is as follows:

From urllib import urlretrievefrom urllib import Urlretrieveurlretrieve (url,save_to_path,show_download_progress)    

As for the URL, simply use the crawler to get the idea. Not the focus of this article, no longer verbose.

The problem is, I don't know why, all the downloaded files are 42k!

Think of the solution, should be a certain limit, the head is not bright, thought for a long while to think, should be the chain of anti-theft.

Check the next, so with Wireshark simulation, found that the header parameters too many, not good each to set (depending on the person's website logical design), so regardless of 3,721, first gave a reffer.

Unfortunate success. Well, no more research, no more work, no more records.

Solutions to the above problems:

Import urllib2req = Urllib2. Request (' http://www.example.com/') req.add_header (' Referer ', ' http://www.python.org/') resp = Urllib2.urlopen (req) Content = Resp.read ()

Next, just need the content to write to the file inside, just a little.

Python download File & anti-theft chain