Python3 Crawler Learning

Learned a few days python3, found that the current learning and crawler still not very related, so now ready to crawl and language synchronization learning.

2016.8.9 Night

From the simplest start, crawl all the contents of the specified URL:

# Encoding:utf-8 Import  "http://www.selflink.cn/selflink"== Data.decode ('  UTF-8')print(data)

# Encoding:utf-8 Import  "http://www.cma.gov.cn/"== Data.decode ('GBK  ')print(data)

Coding is different, one is Utf-8, one is GBK

In addition, when the output is written to a file, if you write an HTML file, open may be garbled, this time do not doubt the Chinese compatibility of Python! (Python Chinese compatibility super good) can actually use Notepad to open a crawl to the file, the encoding is wrong, you can save as a bit, reset the code (such as set to Utf-8), and then open the Web page found not garbled. For example, this code:

ImportUrllib.requesturl="http://www.douban.com/"webpage=urllib.request.urlopen (URL) data=webpage.read () data= Data.decode ('Utf-8') F= Open ("d:/1.html","W") f.write (data)Print(Type (webpage))Print(Webpage.geturl ())Print(Webpage.info ())Print(Webpage.getcode ())

Because Python stores files in ASCII code by default, the file crawled in the browser shows garbled characters. You can manually modify the encoding of the file.

Of course, if you want to set the file encoding automatically, you need to use the codecs library:

Importurllib.requestImportCodecsurl="http://www.douban.com/"webpage=urllib.request.urlopen (URL) data=webpage.read () data= Data.decode ('Utf-8') F= Codecs.open ("d:/1.html","W","Utf-8") f.write (data)Print(Type (webpage))Print(Webpage.geturl ())Print(Webpage.info ())Print(Webpage.getcode ())

Python3 Crawler Learning