CURL-command line download tool

Curl is a file transfer tool that uses URL syntax to work in a command-line manner. It supports many protocols: FTP, FTPS, HTTP, HTTPS, GOPHER, TELNET, DICT, FILE, and LDAP. Curl also supports HTTPS authentication, HTTP POST method, HTTP put method, FTP upload, Kerberos authentication, HTTP upload, proxy server, cookies, username/password Authentication, download file breakpoint, upload file breakpoint, HTTP Proxy Server pipeline (proxy tunneling), even it also supports IPV6, SOCKS5 proxy server, upload files via HTTP proxy server to FTP server and so on, the function is very powerful. Windows operating system under the network ant, the Internet Express (FlashGet) function it can do. To be exact, the Curl branch
File upload and download, so it is a comprehensive transmission tool, but according to the tradition, the user is used to call curl as the download tool.

Curl was developed by the Swedish Curl organization, and you can access http://curl.haxx.se/to obtain its source code and related instructions. Given the extensive use of Curl on Linux, IBM includes this software on the AIX Linux Toolbox CD-ROM, and you can access the IBM Web site http://www-1.ibm.com/servers/aix/products/aixos/ linux/altlic.html download it. The latest version of Curl is a version of 7.9.3 available on the 7.10.8,IBM Web site. The installation under Aix is simple, and the IBM website downloads the RPM package.

In http://curl.haxx.se/docs/, you can download the man Help in UNIX format, which has detailed instructions for the use of Curl tools. The usage of curl is: curl [options] [URL ...] where the options are download the required parameters, about 80 or more, the curl function is entirely dependent on these parameters to complete. The use of specific parameters, users can refer to Curl's man help.

Below, this article will combine concrete examples to explain how to use curl to download.
1. Get a page
Using commands: Curl http://curl.haxx.se
This is the easiest way to use it. Use this command to get the page that http://curl.haxx.se points to, similarly, if the URL here points to a file or a picture can be downloaded directly to the local. If you download an HTML document, the default will not display the file header, the header of the HTML document. To display all, please add the parameter-I, to show only the head, with the parameter-I. At any time, you can use the-v command to see how curl works, and all the commands it sends to the server are displayed. For a continuation of a breakpoint, you can use the-r parameter to specify the transmission range.

2. Access to form (forms)
In Web page design, form is a very important element. A form is typically used to collect and submit information to a Web site. There are two methods of submitting information, get method and post method. Discuss the Get method first, for example in a page:
<form method= "Get" action= "junk.cgi" >
<input type=text name= "Birthyear" >
<input type=submit name=press value= "OK" >
</form>
Then a text box and a button labeled "OK" appear on the browser. By pressing this button, the form submits the data for the text box to the server using the Get method. For example, the original page is seen in www.hotmail.com/when/birth.html, and then you enter 1905 in the text box, and then press the OK button, then the browser URL should now be: "www.hotmail.com/when/ Junk.cgi?birthyear=1905&press=ok "
For such pages, curl can be processed directly, such as to obtain the above page, as long as the input:
Curl "Www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK"
It's OK.
The second way a form is used to submit information is called the Post method, and the difference between the post method and the Get method is that the target URL will be generated in the browser when it is used, and the post will not. Like get, here's a Web page:
<form method= "POST" action= "junk.cgi" >
<input type=text name= "Birthyear" >
<input type=submit name=press value= "OK" >
</form>
A text box and a button labeled "OK" appear on the browser. Press this button to submit the data to the server in the Post method. The URL is not visible at this time, so you need to use a special method to crawl this page:
Curl-d "Birthyear=1905&press=ok" www.hotmail.com/when/junk.cgi
This command can be done.
At the end of 1995, RFC 1867 defined a new post method for uploading files. Mainly used to upload local files to the server. This is how the page is written:
<form method= "POST" enctype= ' multipart/form-data ' action= ' upload.cgi ' >
<input Type=file name=upload>
<input type=submit name=press value= "OK" >
</form>
For this type of page, curl is used differently:
Curl-f upload= @localfilename-F Press=ok [URL]
The essence of this command is to upload the local files to the server using post. There are many uses for post, and users can explore for themselves.

3, use Put method.
The standard method for uploading HTTP protocol files is to use put, when the Curl command uses the-t argument:
Curl-t UploadFile www.uploadhttp.com/receive.cgi

4, the relevant certification.
Curl can handle a variety of conditions of the certification page, such as the download user name/password authentication method of the page (in IE, usually a user name and password input box):
Curl-u Name:password www.secrets.com
If the network is going out through an HTTP proxy server and the proxy server requires a username and password, enter:
Curl-u Proxyuser:proxypassword http://curl.haxx.se
Any time you need to enter a username and password, just specify the username in the parameter and empty the password, curl can interactively let the user enter the password.

5, reference.
Some network resources access must go through another network address jump past, in terms of the term is: referer, reference. For resources of this address, curl can also download:
CURL-E http://curl.haxx.se daniel.haxx.se

6, specify the user client.
Some network resources first need to determine what the user is using the browser, meet the standard to be able to download or browse. At this point curl can "disguise" itself into any other browser:
Curl-a "mozilla/4.0" (compatible; MSIE 5.01; Windows NT 5.0) "[URL]
This directive means that curl is disguised as a IE5.0, and the user platform is Windows 2000. (The other server is based on this string to determine the type of client, so even using AIX does not matter). Use:
Curl-a "mozilla/4.73 [en] (X11; U Linux 2.2.15 i686) "[URL]
At this point Curl became Netscape, running on Linux on the PIII platform.

7. COOKIES
Cookies are a common way for a server to memorize customer information. If the cookie is recorded in the file, use the command:
Curl-b Stored_cookies_in_file www.cookiesite.com
Curl can write a new cookie based on the old cookie and send it to the Web site:
Curl-b cookies.txt-c newcookies.txt www.cookiesite.com

8, the encryption of the HTTP--HTTPS.
If a Web page is transmitted through the OpenSSL encrypted HTTPS protocol, curl can directly access:
Curl https://that.secure.server.com

9, HTTP authentication.
If you are using a certificate-authenticated HTTP address and the certificate is local, then curl uses this:
CURL-E Mycert.pem https://that.secure.server.com

Reference books and notes: Curl is very broad, users want to use this tool, in addition to detailed learning parameters, but also need to understand the various protocols of HTTP and the various syntax of the URL. Several readings are recommended here:
RFC 2616 the definition of HTTP protocol syntax.
RFC 2396 the definition of URL syntax.
RFC 2109 how cookies work.
RFC 1867 http How to post, and the format of post