Examples of urllib module usages in Python _python

This example describes the use of Urllib modules in Python. Share to everyone for your reference. The specific analysis is as follows:

First, the question:

The recent requirements of the company project is based on the customer provided by the API, we regularly go to get data, the previous program is to use the PHP collection task into the Redis queue, and then in Linux to do a resident process run a php file, the php file on an infinite loop, Judge Redis queue, have on the implementation , not on the break.

Second, the solution:

Just learned about Python recently, Python's Urllib module may be faster and simpler than PHP's curl. Put a little bit of code

Copy Code code as follows:

#_ *_ Coding:utf-8 _*_
Import Sys
Reload (SYS)
Sys.setdefaultencoding ("Utf-8")
Import OS
Import JSON
From Urllib import Urlopen
doc = Urlopen ("http://xxxx?webid=1&tid=901&cateid=101"). Read ()
doc = Json.loads (doc)
Print doc
Print Doc.keys ()
Print doc["MSG"]
Print doc[' data ']
Print doc[' ret ']

The time required to discover the first visit is [finished in 3.0s]
And the time for the second visit is [finished in 0.2s]
The Python urllib module is visible with a cache
Typical examples of URLLIB/2 usage

Copy Code code as follows:

Import Urllib2
Import Cookielib
Import Urllib

Class Hi_login:
def __init__ (self):
Cookie = Cookielib. Cookiejar ()
Self.cookie = Urllib2. Httpcookieprocessor (cookie) ##### Generate Cookies ###

def login (self,user,pwd):
Url= ' Http://passport.baidu.com/?login '
Postdata=urllib.urlencode ({
' Mem_pass ': ' On ',

' Password ':p WD
' Submit ': ',
' TPL ': ' SP ',
' Tp_reg ': ' SP ',
' U ': ' http://hi.baidu.com ',
' username ': User})
### Proxy_support = urllib2. Proxyhandler ({"http": "http://ahad-haam:3128"}) and then join the opener method
Opener = Urllib2.build_opener (Self.cookie) ### using Cookies ###
headers = {####### dict structure, can join x-forward-for or even refer, etc. #######
' User-agent ': ' mozilla/5.0 (Windows; U Windows NT 6.1; En-us; rv:1.9.1.6) gecko/20091201 firefox/3.5.6 '}
Urllib2.install_opener (opener)
Request = Urllib2. Request (Url,urllib.urlencode (postdata), headers = headers)
Urllib2.urlopen (Request)
If __name__== ' __main__ ':
Pwd= ' 123456 '
User= ' Xiaofu '
Test=hi_login ()
Test.login (USER,PWD)

If you visit a page that requires authentication, such as a Nagios monitor page,

Copy Code code as follows:

Import Urllib2
Password_mgr = Urllib2. Httppasswordmgrwithdefaultrealm ()
url = "Http://202.1.x.y/nagios"
Password_mgr.add_password (None, URL, user= ' abc ', passwd= ' xxxxxx ')
Handler = Urllib2. Httpbasicauthhandler (Password_mgr)
Opener = Urllib2.build_opener (handler)
Urllib2.install_opener (opener)
F=urllib2.urlopen (URL)
Print F.code

Return result 200, or 401 authentication error

I hope this article will help you with your Python programming.