Python 3.4 crawler, camouflage browser (403 Forbidden)

When you use Python to crawl Web images, you occasionally encounter a 403 error. This may be because the server banned the crawler. In this case, if you want to continue to crawl the picture, you need to add header information in the request, disguised as a browser.

If you're using the python3.4 version, it might be a bit of a hassle if you want to find a way to add a header to your request online. After some experimentation, it is possible to determine the following code.

" "Created on 2014-9-23@author:holiday" "Importurllib.requestImportOsurl="http://img0.imgtn.bdimg.com/it/u=4054848240,1657436512&fm=21&gp=0.jpg"#headers = [(' User-agent ', ' mozilla/5.0 (Windows NT 6.1) applewebkit/537.11 (khtml, like Gecko) chrome/23.0.1271.64 Saf ari/537.11 '),#(' Accept ', ' text/html;q=0.9,*/*;q=0.8 '),#(' Accept-charset ', ' iso-8859-1,utf-8;q=0.7,*;q=0.3 '),#(' accept-encoding ', ' gzip '),#(' Connection ', ' close '),#(' Referer ', None)] #注意如果依然不能抓取的话, here you can set the host of the crawl siteheaders = [('Host','img0.imgtn.bdimg.com'),('Connection','keep-alive'),('Cache-control','max-age=0'),('Accept','text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'),('user-agent','mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/36.0.1985.125 safari/537.36'),('accept-encoding','GZIP,DEFLATE,SDCH'),('Accept-language','zh-cn,zh;q=0.8'),('If-none-match','90101f995236651aa74454922de2ad74'),('Referer','http://image.baidu.com/i?tn=baiduimage&ps=1&ct=201326592&lm=-1&cl=2&nc=1&word=%E4% Ba%a4%e9%80%9a&ie=utf-8'),('if-modified-since','Thu, 1970 00:00:00 GMT')]opener=Urllib.request.build_opener () opener.addheaders=Headersdata=opener.open (URL)#print (data)Path="c:/151.jpg"F= Open (Path,"WB") F.write (Data.read ()) F.close ()

Thankfully, it's also part of the download. The images you visit are saved in 151.jpg under the C drive.

It is important to note that

('Referer','http://image.baidu.com/i?tn=baiduimage&ps=1&ct= 201326592&lm=-1&cl=2&nc=1&word=%e4%ba%a4%e9%80%9a&ie=utf-8'),

This line of code in the header tells the server that I'm jumping over from this page. (Which of course is not true) and then cheated the server.

From the beginning, because of this sentence has not been added has not been successful.

Python 3.4 crawler, camouflage browser (403 Forbidden)