Curl Website Development Guide

Curl Website Development Guide

Nanyi

Date: September 4, 2011

I always thought that curl was just a library of functions for programming.

Recently discovered that this command itself, is a very useful web development tool, please see my collation of its usage.

===================================

Curl Website Development Guide

Nanyi Finishing

Curl is a command-line tool that sends a network request and then gets and extracts the data, which is displayed on the "standard output" (stdout).

It supports a variety of protocols, and the following examples explain how to use it for web development.

First, view the source of the Web page

Directly after the Curl command add a URL, you can see the Web source. We take the URL www.sina.com as an example (select the URL, mainly because its page code is shorter):

Curl www.sina.com

<! DOCTYPE HTML PUBLIC "-//ietf//dtd HTML 2.0//en" >
<title>301 Moved permanently</title>
<p>the document has moved <a href= "http://www.sina.com.cn/" >here</a>.</p>
</body>

If you want to save this page, you can use the-o parameter, which is equivalent to using the wget command.

Curl-o [file name] Www.sina.com

Second, automatic jump

Some URLs are automatically redirected. With the-l parameter, curl jumps to the new URL.

Curl-l www.sina.com

Type the above command, and the result will automatically jump to www.sina.com.cn.

Third, display the head information

The-I parameter can display the header information of the HTTP response, along with the page code.

Curl-i www.sina.com

http/1.0 301 Moved permanently
Date:sat, SEP 2011 2 　　3:44:10 GMT
server:apache/2.0.54 (Unix)
location:http://www.sina.com.cn/
cache-control:max-age=3600
Expires:sun, Sep 00:44:10 GMT
vary:accept-encoding
content-length:231
content-type:text/html; Charset=iso-8859-1
age:3239
X-cache:hit from sh201-9.sina.com.cn
connection:close

<! DOCTYPE HTML PUBLIC "-//ietf//dtd HTML 2.0//en";
<title>301 Moved Permanently </title>
<p>the Document Has moved <a href= "http://www.sina.com.cn/" >HERE</A>.</P>
</body>

The-I parameter shows only the header information for the HTTP response.

Iv. Display of communication processes

The-v parameter can display the entire process of an HTTP communication, including port connections and HTTP request header information.

Curl-v www.sina.com

* About-to-connect () to www.sina.com Port (#0)
* Trying 61.172.201.195 ... Connected
* Connected to Www.sina.com (61.172.201.195) port (#0)
> get/http/1.1
> user-agent:curl/7.21.3 (I686-PC-LINUX-GNU) libcurl/7.21.3 openssl/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host:www.sina.com
> Accept: */*
>
* HTTP 1.0, assume close after body
< http/1.0 301 Moved Permanently
< Date:sun, Sep 00:42:39 GMT
< server:apache/2.0.54 (Unix)
< location:http://www.sina.com.cn/
< cache-control:max-age=3600
< Expires:sun, Sep 01:42:39 GMT
< vary:accept-encoding
< content-length:231
< content-type:text/html; Charset=iso-8859-1
< X-cache:miss from sh201-19.sina.com.cn
< Connection:close
<
<! DOCTYPE HTML PUBLIC "-//ietf//dtd HTML 2.0//en" >
<title>301 Moved permanently</title>
<p>the document has moved <a href= "http://www.sina.com.cn/" >here</a>.</p>
</body>* Closing Connection #0

If you think the above information is not enough, then the following command can see a more detailed communication process.

Curl--trace output.txt www.sina.com

Or

Curl--trace-ascii output.txt www.sina.com

After you run, open the Output.txt file to view.

V. Send form Information

Send form information with Get and post two methods. The Get method is relatively simple, as long as the data is attached to the URL behind the line.

Curl Example.com/form.cgi?data=xxx

The Post method must separate the data from the URL, and curl will use the--data parameter.

Curl--data "Data=xxx" example.com/form.cgi

If your data is not encoded by the form, you can also have curl encode it for you, with the parameter--data-urlencode.

Curl--data-urlencode "Date=april 1" example.com/form.cgi

Vi. http verbs

The default HTTP verb for curl is get, which uses the-X parameter to support other verbs.

Curl-x POST www.example.com

Curl-x DELETE www.example.com

Vii. File Upload

Assume that the file is uploaded by the following form:

<form method= "POST" enctype= ' Multipart/form-data ' action= "upload.cgi" >
<input Type=file name=upload>
<input type=submit name=press value= "OK" >
</form>

You can use Curl to upload files like this:

Curl--form [email protected]--form Press=ok [URL]

Viii.. referer Field

Sometimes you need to provide a referer field in the HTTP request header to indicate where you are jumping from.

Curl--referer http://www.example.com http://www.example.com

Ix. User Agent Field

This field is used to represent the device information for the client. Depending on the field, the server sometimes returns pages of different formats, such as mobile and desktop versions, for different devices.

The user agent for IPhone4 is

mozilla/5.0 (IPhone; U CPU iPhone os 4_0 like Mac os X; En-US) applewebkit/532.9 (khtml, like Gecko) version/4.0.5 mobile/8a293 safari/6531.22.7

Curl can simulate this:

Curl--user-agent "[user agent]" [URL]

X. Cookies

Using the--cookie parameter, you can let curl send a cookie.

Curl--cookie "Name=xxx" www.example.com

As for the value of the specific cookie, it can be obtained from the Set-cookie field of the HTTP response header information.

Xi. Add header information

Sometimes it is necessary to add a header message to the HTTP request itself. The--header parameter can play this role.

Curl--header "Content-type:application/json" http://example.com

12. HTTP Authentication

Some domains require HTTP authentication, and curl needs to use the--user parameter.

Curl--user Name:password example.com

Resources

* Using CURL to automate HTTP jobs

* Teach you to learn to use curl

* 9 uses for CURL worth knowing

Curl Website Development Guide