Example of Urllib module usage in python

This example describes the use of the Urllib module in Python. Share to everyone for your reference. The specific analysis is as follows:

First, the question:

The recent company project requirements are based on customer-provided API, we regularly go to get data, the previous plan is to use PHP collection tasks into the Redis queue, and then a permanent process under Linux to run a php file, the php file on an infinite loop, determine the Redis queue, there is the execution , there is no break.

Second, the solution:

Recently just learned Python, Python's urllib module may be faster and simpler than PHP's curl. Paste the code.
Copy the Code code as follows:

#_ *_ Coding:utf-8 _*_
Import Sys
Reload (SYS)
Sys.setdefaultencoding ("Utf-8")
Import OS
Import JSON
From Urllib import Urlopen
doc = Urlopen ("http://xxxx?webid=1&tid=901&cateid=101"). Read ()
doc = Json.loads (doc)
Print doc
Print Doc.keys ()
Print doc["MSG"]
Print doc[' data '
Print doc[' ret ']

The time required to find the first visit is [finished in 3.0s]
The second access time is [finished in 0.2s]
The Python urllib module is visible with a cached
Typical examples of URLLIB/2 usage
Copy CodeThe code is as follows:

Import Urllib2
Import Cookielib
Import Urllib

Class Hi_login:
def __init__ (self):
Cookie = Cookielib. Cookiejar ()
Self.cookie = Urllib2. Httpcookieprocessor (cookie) ##### generate Cookie # # #

def login (self,user,pwd):
Url= ' Http://passport.baidu.com/?login '
Postdata=urllib.urlencode ({
' Mem_pass ': ' On ',

' Password ':p WD
' Submit ': ',
' TPL ': ' SP ',
' Tp_reg ': ' SP ',
' U ': ' http://hi.baidu.com ',
' username ': User})
# # # Proxy_support = urllib2. Proxyhandler ({"http": "http://ahad-haam:3128"}) then join the opener method
Opener = Urllib2.build_opener (Self.cookie) # # # Use cookies # # #
headers = {####### dict structure, can join x-forward-for even refer etc. #######
' User-agent ': ' mozilla/5.0 (Windows; U Windows NT 6.1; En-us; rv:1.9.1.6) gecko/20091201 firefox/3.5.6 '}
Urllib2.install_opener (opener)
Request = Urllib2. Request (Url,urllib.urlencode (postdata), headers = headers)
Urllib2.urlopen (Request)
If __name__== ' __main__ ':
Pwd= ' 123456 '
User= ' Xiaofu '
Test=hi_login ()
Test.login (USER,PWD)

If you visit a page that requires authentication, such as the Nagios monitoring page,
Copy CodeThe code is as follows:

Import Urllib2
Password_mgr = Urllib2. Httppasswordmgrwithdefaultrealm ()
url = "Http://202.1.x.y/nagios"
Password_mgr.add_password (None, URL, user= ' abc ', passwd= ' xxxxxx ')
Handler = Urllib2. Httpbasicauthhandler (Password_mgr)
Opener = Urllib2.build_opener (handler)
Urllib2.install_opener (opener)
F=urllib2.urlopen (URL)
Print F.code

Return result 200, otherwise 401 authentication error

Hopefully this article will help you with Python programming.

