Urllib Learning notes in Python

Just learned about Python recently, Python's Urllib module may be faster and simpler than PHP's curl. Put a little bit of code

The code is as follows	Copy Code
#_ *_ Coding:utf-8 _*_ Import Sys Reload (SYS) Sys.setdefaultencoding ("Utf-8") Import OS Import JSON From Urllib import Urlopen doc = Urlopen ("http://xxxx?webid=1&tid=901&cateid=101"). Read () doc = Json.loads (doc) Print doc Print Doc.keys () Print doc["MSG"] Print doc[' data '] Print doc[' ret ']

The time required to discover the first visit is [finished in 3.0s]

And the time for the second visit is [finished in 0.2s]

The Python urllib module is visible with a cache

Typical examples of URLLIB/2 usage

The code is as follows

Copy Code

Import Urllib2 Import Cookielib Import Urllib Class Hi_login: def __init__ (self): Cookie = Cookielib. Cookiejar () Self.cookie = Urllib2. Httpcookieprocessor (cookie) ##### Generate Cookies ### def login (self,user,pwd): Url= ' Http://passport.baidu.com/?login ' Postdata=urllib.urlencode ({ ' Mem_pass ': ' On ', ' Password ':p WD ' Submit ': ', ' TPL ': ' SP ', ' Tp_reg ': ' SP ', ' U ': ' http://hi.baidu.com ', ' username ': User}) ### Proxy_support = urllib2. Proxyhandler ({"http": "http://ahad-haam:3128"}) and then join the opener method Opener = Urllib2.build_opener (Self.cookie) ### using Cookies ### headers = {####### dict structure, can join x-forward-for or even refer, etc. ####### ' User-agent ': ' mozilla/5.0 (Windows; U Windows NT 6.1; En-us; rv:1.9.1.6) gecko/20091201 firefox/3.5.6 '} Urllib2.install_opener (opener) Request = Urllib2. Request (Url,urllib.urlencode (postdata), headers = headers) Urllib2.urlopen (Request) If __name__== ' __main__ ': Pwd= ' 123456 ' User= ' Xiaofu ' Test=hi_login () Test.login (USER,PWD)

If you visit a page that requires authentication, such as a Nagios monitor page,

The code is as follows	Copy Code
Import Urllib2 Password_mgr = Urllib2. Httppasswordmgrwithdefaultrealm () url = "Http://202.1.x.y/nagios" Password_mgr.add_password (None, URL, user= ' abc ', passwd= ' xxxxxx ') Handler = Urllib2. Httpbasicauthhandler (Password_mgr) Opener = Urllib2.build_opener (handler) Urllib2.install_opener (opener) F=urllib2.urlopen (URL) Print F.code

Return result 200, or 401 authentication error