Use curl to detect each stage of an HTTP request initiated by the Clinet side

Objective

Last week, one of our customer feedback, access to our interface, the average time spent on more than 2s. However, we have to monitor the request, and the request to return, the whole process is monitored, our time is basically within 50ms, very fast.

Later learned that customers from Guangdong visit to our Shanghai, the public network to visit us. Then it is recommended to detect, DNS time-consuming, TCP setup time-consuming and so on. Theoretically, the long-distance public network, the network delay is very high. It is recommended to use curl to check. As you can guess, the TCP setup takes a long time.

To get to the point, this article focuses on using curl to detect the time of each phase of an HTTP request initiated by the client side.

First, the process of the HTTP request is introduced

An HTTP request that involves multiple stages

1, DNS resolution domain name

2. Request routing from clinet to server,clinet to establish a TCP connection with server

3, if the use of HTTPS, also involves the establishment of SSL connection

4. The server begins to prepare the data

Start logical calculation, back-end interface, check database cache, etc.

5. Server begins to pass data

Data ready, start data transfer to client

6. Data transfer complete

7, the entire process may also involve multiple redirects

Second, the introduction of curl

Curl is an open source data transfer tool that uses URL syntax to work in command line mode.

Support: DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, Pop3s, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SM TP, SMTPS, Telnet and TFTP. Curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, HTTP/2, cookies, user +password Authentication (Basic, Plain, Digest, Cram-md5, NTLM, Negotiate and Kerberos), file transfer resume, proxy Tunne Ling, etc.

Latest version of Curl stable version 7.55.1 (as of 20170817)

Source code: Https://github.com/curl/curl

Third: Use curl to detect the clinet side of the HTTP request each phase of the time, a brief description

650) this.width=650; "Src=" https://s1.51cto.com/wyfs02/M00/9E/C1/wKioL1mVjoSRgA_ZAAGP7R3orEs850.png-wh_500x0-wm_ 3-wmp_4-s_3719930826.png "title=" Curl2.png "style=" Float:none; "alt=" wkiol1mvjosrga_zaagp7r3ores850.png-wh_50 "/ >

1. TCP time to establish a connection: Connect-namelookup

2. Establish a TCP connection to the time that the server returns the first byte of the client:

Starttransfer-connect

3. When the server processes the data:

can be calculated with Starttransfer-pretransfer

4, the client receives the data time-consuming (starts receives to receive completes):

Total-starttransfer

Iv. Examples:

Curl-o/dev/null-s-w time_namelookup: "\ T"%{time_namelookup} "\ n" time_connect: "\t\t"%{time_connect} "\ n" time_ AppConnect: "\ T"%{time_appconnect} "\ n" time_pretransfer: "\ T"%{time_pretransfer} "\ n" time_starttransfer: "\ T"%{time _starttransfer} "\ n" time_total: "\t\t"%{time_total} "\ n" time_redirect: "\t\t"%{time_redirect} "\ n" https:// www.yqb.com/

650) this.width=650; "Src=" https://s5.51cto.com/wyfs02/M02/9E/C1/wKioL1mVjobzldy3AAEPmtIEJNQ613.png-wh_500x0-wm_ 3-wmp_4-s_2512627456.png "title=" Curl3.png "style=" Float:none; "alt=" wkiol1mvjobzldy3aaepmtiejnq613.png-wh_50 "/ >

1. DNS Parsing time: 0.008s

2. TCP time to establish a connection: 0.059-0.008=0.051s

3. The SSL handshake takes time to complete : 0.228-0.059=0.169s

169ms, a layer of SSL is still time-consuming

4. When the server processes the data: 0.287-0.228=0.059

59ms, indicating that the server is processing quickly

5, the overall time-consuming: 0.228s

6, the whole process is not redirect , so the redirect time is 0

One more example:

650) this.width=650; "Src=" https://s4.51cto.com/wyfs02/M02/00/12/wKiom1mVjomxw8_qAACQJsxBKgc809.png-wh_500x0-wm_ 3-wmp_4-s_3139750192.png "title=" Curl4.png "style=" Float:none; "alt=" wkiom1mvjomxw8_qaacqjsxbkgc809.png-wh_50 "/ >

Time consuming for the server to process data: 2.305-0.014=2.291s

This means that the performance of the server side is worth investigating.

For the server side, it is time-consuming to analyze it:

Cache and DB for load balancing, firewall---

Need to go deep to analyze the time spent in which link, targeted optimization.

Five, detailed description

Namelookup: from the start, the domain name resolution is time-consuming to complete

Curlinfo_namelookup_time . The time it took from the start until the name resolving is completed.

CONNECT : From the start, the TCP setup time is complete

Curlinfo_connect_time . The time it took from the start until the connect to the remote host (or proxy) is completed.

AppConnect : From the beginning, the application layer (SSL, the application layer above TCP) connection/handshake completion time

Curlinfo_appconnect_time . The time it took from the start until the SSL connect/handshake with the remote host is completed. (Added in 7.19.0)

Pretransfer: time-consuming to start transferring data from the beginning of the calculation

Curlinfo_pretransfer_time . The time it took from the start until the file transfer was just to begin. This includes all Pre-transfer commands and negotiations that is specific to the particular protocol (s) involved.

Starttransfer: The time taken to start transferring data from the beginning of the calculation (Libcurl receives the first byte)

Curlinfo_starttransfer_time . The time it took from the start until the first byte was received by Libcurl.

Total: total time-consuming

Curlinfo_total_time . total time of the previous request.

REDIRECT: The duration of the entire process redirection, if the entire process is not redirected, this time is 0

Curlinfo_redirect_time . The time it took for all redirection steps include name lookup, connect, Pretransfer and transfer before final transaction was started. So, this is the zero if no redirection took place.

Another: Python also has a Pycurl module, everyone can try.

Reference:

Https://curl.haxx.se/libcurl/c/curl_easy_getinfo.html

This article is from "H2O's Operation & Development Road" blog, reprint please contact the author!

Use curl to detect each stage of an HTTP request initiated by the Clinet side