Detecting Web Server Quality--pycurl

Pycurl is a C language written in Libcurl python implementation, the function is very powerful, supporting the operation of the Protocol has FTP, HTTP, HTTPS, Telnet, etc., can be understood as the Linux under the Curl command function Python package, easy to use.

The following will be done by invoking the method provided by Pycurl to achieve the quality of Web services, such as the corresponding HTTP status code, request delay, HTTP header information, download speed, etc., using this information can be used to locate the service response slow specific links.

First, the module commonly used method description

Pycurl. The curl () class implements a Libcurl package that creates a curl handle object with no parameters, and the following describes several common methods of Curl objects.

(1) Close (): corresponding to the Libcurl package in the Curl_easy_cleanup method, no parameters, to achieve close, recovery curl cashing.

(2) Perform (): Corresponds to the Curl_easy_perform method in the Libcurl package, without parameters, to implement the Curl object request submission.

(3) setopt (Option,value): Corresponds to the Curl_easy_setopt method in the Libcurl package, the parameter option is specified by a constant of Libcurl, the value of the parameter value is dependent on option, can be a string, an integer type , file objects, lists, functions, and so on, here are some common constants:

In [1]:ImportPycurl#Import ModuleIn [2]: c = pycurl. Curl ()#Create a Curl objectIn [3]: c.setopt (pycurl. connecttimeout,5)#the wait time for the connection, set to 0 does not waitIn [4]: C.setopt (pycurl. timeout,5)#request time-out periodIn [5]: c.setopt (pycurl. noprogress,0)#whether to block the download progress bar, not 0 maskingIn [6]: c.setopt (pycurl. maxredirs,5)#Specify the maximum number of HTTP redirectsIn [7]: c.setopt (pycurl. forbid_reuse,1)#Disconnect after completion of interaction, no reuseIn [8]: c.setopt (pycurl. fresh_connect,1)#force a new connection, which is the replacement of a connection in the cacheIn [9]: c.setopt (pycurl. DNS_CACHE_TIMEOUT,60)#set the time to save DNS information by default of 120 secondsIn [ten]: C.setopt (pycurl. Url'http://www.baidu.com')#request the specified URLIn [one]: c.setopt (pycurl. UserAgent,'mozilla/5.2 (compatible; MSIE 6.0; Windows NT 5.1;sv1;. NET CLR 1.1.4322;. NET CLR 2.0.50324)')#Configuring the User-agent of the request HTTP headerIn []: C.setopt (pycurl. Headerfunction, GetHeader)#directs the returned HTTP header to the callback function GerheaderIn []: C.setopt (pycurl. Writefunction, GetBody)#directs the returned content to the callback function GetBodyIn []: C.setopt (pycurl. Writeheader, Fileobj)#directs the returned HTTP header to the Fileobj fileIn []: C.setopt (pycurl. WriteData, Fileobj)#directs the returned HTML content to the Fileobj file object

(4) GetInfo (option): Corresponds to the Curl_easy_getinfo method in the Libcurl package, the parameter option is specified by the Libcurl constant. A list of commonly used constants is listed below:

In [1]:ImportPycurlin [2]: c = pycurl. Curl ()#Create a Curl objectIn [3]: C.getinfo (pycurl. HTTP_CODE)#return HTTP status codeIn [4]: C.getinfo (pycurl. Total_time)#total time consumed by transfer endIn [5]: C.getinfo (pycurl. Namelookup_time)#time consumed by DNS resolutionIn [6]: C.getinfo (pycurl. Connect_time)#time spent establishing a connectionIn [7]: C.getinfo (pycurl. Pretransfer_time)#the time consumed from establishing the connection to preparing the transferIn [8]: C.getinfo (pycurl. Starttransfer_time)#the time from when the connection was established to the start of the transferIn [9]: C.getinfo (pycurl. Redirect_time)#time spent in redirectionIn [ten]: C.getinfo (pycurl. Size_upload)#size of the upload packetIn [one]: C.getinfo (pycurl. Size_download)#size of the download packetIn []: C.getinfo (pycurl. Speed_download)#Average Download speedIn []: C.getinfo (pycurl. Speed_upload)#Average upload speedIn []: C.getinfo (pycurl. Header_size)#HTTP Header Size

We use these constant values provided by the Libcurl package to achieve the purpose of detecting Web service quality.

Second, practice: to realize the quality of Web service detection

HTTP service is one of the most popular Internet applications, the quality of service is related to the user experience and the operation of the site service level, the most commonly used there are two standards,

The availability of a service, such as whether it is in a normal service state, rather than 404 Page not found or 500 page error, etc.;

The second is the response speed of the service, such as the static class file download time is controlled in the millisecond level, dynamic CGI is the second level.

This example uses Pycurl's setopt and GetInfo methods to implement HTTP quality of service probes.

#!/usr/bin/env Python2#Coding:utf-8ImportOs,sysImport TimeImportPycurlurl="http://www.google.com"  #target URL for probingc = Pycurl. Curl ()#Create a Curl objectC.setopt (Pycurl. Url,url)#define URL constants for requestsC.setopt (Pycurl. connecttimeout,5)#to define the wait time for a request connectionC.setopt (Pycurl. timeout,5)#Define request time-outsC.setopt (Pycurl. noprogress,1)#block Download progress barC.setopt (Pycurl. maxredirs,1)#Specifies the maximum number of HTTP redirects is 1C.setopt (Pycurl. forbid_reuse,1)#Force Disconnect after completion of interaction, no reuseC.setopt (Pycurl. dns_cache_timeout,30)#setting the time to save DNS information is 30 seconds#creates a file object, opened as "WB" to store the contents of the returned HTTP headers and pagesIndexfile = open (Os.path.dirname (Os.path.realpath (__file__))+"/content.txt","WB") c.setopt (Pycurl. Writeheader,indexfile)#directs the returned HTTP header to the Indexfile fileC.setopt (Pycurl. Writedata,indexfile)#directs the returned HTML content to the Indexfile fileTry: C.perform ()exceptexception,e:Print "Connection Error"indexfile.close () c.close () sys.exit () namelookup_time= C.getinfo (C.namelookup_time)#time consumed by DNS resolutionConnect_time = C.getinfo (c.connect_time)#time spent establishing a connectionPretransfer_time = C.getinfo (c.pretransfer_time)#the time consumed from establishing the connection to preparing the transferStarttransfer_time = C.getinfo (c.starttransfer_time)#the time from when the connection was established to the start of the transferTotal_time = C.getinfo (c.total_time)#total time consumed by transfer endHttp_code = C.getinfo (C.http_code)#return HTTP status codeSize_download = C.getinfo (c.size_download)#size of the download packetHeader_size = C.getinfo (c.header_size)#HTTP Header SizeSpeed_download = C.getinfo (c.speed_download)#Average Download speed#print output related dataPrint "HTTP status code:%s"%Http_codePrint "DNS resolution time:%2F"% namelookup_time*1000Print "established connection time:%2f"% connect_time*1000Print "Ready transfer time:%2f"% preteansfer_time*1000Print "Transfer start time:%2f"% starttransfer_time*1000Print "total time to transfer:%2F"% total_time*1000Print "Download Packet size:%d bytes/s"%Size_downloadPrint "HTTP Header size:%d bytes/s"%header_sizePrint "average download speed:%d bytes/s"%Speed_download#close files and Curl objectsindexfile.close () c.close ( )

Detecting Web Server Quality--pycurl