Perfect use of Curl under Linux to access Web pages

First of all, you have to learn the function of the various parameters under curl, can help you to a large extent, often need dozens of lines or even hundreds of lines of code, really only need two or three lines of command code to get everything done!

-a/--append  upload file, attach to target file  -A/--user-agent <string>   simulate browser login server  -  anyauth    can use "any" authentication method  -b/--cookie <name=string/file> cookie string or file read location  - basic  using HTTP Basic authentication  -B/--use-ascii  using ascii /text transfer  -c/--cookie-jar < file>  the cookie to this file after the end of the Operation  -C/--continue-at <offset>   Breakpoint continuation  -d/--data  <data>   http post Way to transfer data  --data-ascii <data>   Post data in ASCII mode  --data-binary <data>  post data in binary mode  --negotiate       using HTTP authentication  --digest         using digital authentication  -- disable-eprt   prohibit the use of eprt or lprt --disable-epsv   prohibit the use of epsv -d/--dump-header < file>  writes header information to the file  --egd-file <file>  sets Egd socket path for random data (SSL)  -- tcp-nodelay    using the Tcp_nodelay option -e/--referer  Source URL  -e/--cert <cert[:p asswd]>  client certificate file and password   (SSL)  -- cert-type <type>  certificate file type   (Der/pem/eng)   (SSL)  --key <key>       private key filename   (SSL)  --key-type <type>  private key file type   (Der/pem/eng)   (SSL)  --pass  <pass>   private Key Password   (SSL)  --engine <eng>   Encryption engine uses   (SSL) .  "--engine list" &NBSP;FOR&NBSP;LIST&NBSP;--CACERT&NBSP;&LT;FILE&GT;&NBSP;CA Certificate   (SSL)  --capath <directory> ca directory   (Made using c_rehash)  to  verify peer against  (SSL)  --ciphers <list>  ssl password  --compressed      requirements return is compressed in the situation   (Using deflate or gzip)  --connect-timeout  <seconds>  set maximum request time  --create-dirs    Create a directory hierarchy of local directories  --crlf            upload is to convert LF into crlf -f/--fail           Do not display HTTP error when connection fails  --ftp-create-dirs  create remote directory If remote directory does not exist &NBSP;--FTP-METHOD&NBSP;[MULTICWD/NOCWD/SINGLECWD]   Controlling the use of CWD  --ftp-pasv       using  PASV/EPSV  instead of Port  -- ftp-skip-pasv-ip  when using PASV, ignore the IP address  --ftp-ssl        try to use  ssl/ tls  FTP data transfer  --ftp-ssl-reqd   require  SSL/TLS  for FTP data transfer  -F/--form < name=content>  analog HTTP form submission Data  -form-string <name=string>  analog HTTP form submission data  -g/-- globoff  disabling URL sequences and ranges use {} and [] -g/--get  to send data in Get- -h/--help  help  -H/--header  <line> Custom header information passed to server  --ignore-content-length   ignored length of HTTP header information  -i/--include  Include protocol header information at output  -I/--head   Show only Document information   read from file-j/--junk-session-cookies ignore session cookie -  Interface <interface> Specify network interface/address using &NBSP;-&NBSP;KRB4&NBsp;< level > Enable krb4 -j/--junk-session-cookies  read files with the specified security level ignore Session cookie --interface  <interface>  using the specified network interface/address  --krb4 <level>   krb4 -k/--with the specified security level insecure  allows you to not use certificates to SSL sites  -K/--config   specified profile reads  -l/--list-only  lists file names under the FTP directory   --limit-rate <rate>  Set transfer speed  --local-port<NUM>  enforce use of local port number  -m/--max-time  <seconds>  set Maximum transfer time  --max-redirs <num>  set the maximum number of read directories  --max-filesize  <bytes>  set the maximum amount of files to download  -M/--manual   show full manual  -n/--netrc  read user names and passwords from the Netrc file  --netrc-optional  use  .netrc  or  url to cover-n --ntlm            Use  HTTP NTLM  authentication  -N/--no-buffer  disable buffered output  -o/-- output  writes the output to the file  -O/--remote-name  writes the output to the file, preserving the file name of the remote files  -p/--proxytunnel    Use HTTP proxy  --proxy-anyauth  SelectChoose either proxy authentication method  --proxy-basic    use Basic authentication on the proxy  --proxy-digest   use digital authentication on the proxy   --proxy-ntlm     using NTLM authentication  -P/--ftp-port <address>  port address on the agent, Instead of using pasv -q/--quote <cmd> file transfer, send command to server  -r/--range <range> retrieve from http/ 1.1 or FTP server byte range  --range-file  read (SSL) random files  -R/--remote-time    Preserve remote file time when generating files locally  --retry <num>    number of retries when there is a problem with transmission  --retry-delay <seconds>    When a problem occurs with transmission, set the maximum retry time  -s/--silent silent mode when setting the retry interval  --retry-max-time <seconds>  transmission problem. Do not output anything  -S/--show-error    display error  --socks4 
Two, commonly used curl instance
1, crawl the page content into a file
[Email protected] mytest]# Curl-o home.html http://www.codesky.net
2, with-O (uppercase), the following URL to specific to a file, or not catch it down. We can also use the regular to grab things.
[Email protected] mytest]# Curl-o Http://www.codesky.net/wp-content/uploads/2010/09/compare_varnish.jpg[[email Protected] mytest]# Curl-o http://www.codesky.net/wp-content/uploads/2010/[0-9][0-9]/aaaaa.jpg
3, simulate form information, simulate login, save cookie Information
[Email protected] mytest]# curl-c/cookie_c.txt-f log=aaaa-f pwd=****** http://www.codesky.net/wp-login.php
4, simulating form information, simulating login, saving header information
[Email protected] mytest]# curl-d/cookie_d.txt-f log=aaaa-f pwd=****** http://www.codesky.net/wp-login.php
The cookie generated by-C (lowercase) is not the same as the cookie in-D.
5, use cookie file
[Email protected]gcms01 mytest]# curl-b./cookie_c.txt http://www.codesky.net/wp-admin
6, the breakpoint continues to pass,-C (uppercase)
[Email protected] mytest]# curl-c-o http://www.codesky.net/wp-content/uploads/2010/09/compare_varnish.jpg
7, transfer the data, it is best to use the login page test, because you pass the value of the past, curl back to grasp the data, you can see that you pass the value has no success
[Email protected] mytest]# curl-d log=aaaa http://www.codesky.net/wp-login.php
8, show the fetch error, the following example, it is very clear that.
[Email protected] mytest]# curl-f Http://www.codesky.net/asdfcurl: (+) The requested URL returned Error:404[[email Pro Tected] mytest]# Curl http://www.codesky.net/asdf<HTML><HEAD><TITLE>404,not found</title> ... 。。。。
9, forged source address, some website will judge, request source address.
[Email protected] mytest]# curl-e http://localhost http://www.codesky.net/wp-login.php
10, when we often use curl to engage in people's things, people will put your IP to shield off, this time, we can use the agent
[Email protected] mytest]# curl-x 24.10.28.84:32779-o home.html http://www.codesky.net
11, than the larger stuff that we can download in sections
[[email protected] mytest]# curl -r 0-100 -o img.part1 http:// www.codesky.net/wp-content/uploads/2010/09/compare_varnish.jpg % Total    %  Received % Xferd  Average Speed   Time     Time     time  current dload  upload   total    spent    left  speed100   101  100    101    0     0    105       0 --:--:-- --:--:-- --:--:--     0[[ email protected] mytest]# curl -r 100-200 -o img.part2 http:// www.codesky.net/wp-content/uploads/2010/09/compare_varnish.jpg % Total    %  received % xferd  average speed   time    time     time   Current Dload  Upload   Total   Spent     Left  Speed100   101  100   101     0     0     57      0   0:00:01  0:00:01 --:--:--     0[[email protected]  mytest]# curl -r 200- -o img.part3 http://www.codesky.net/wp-content/ Uploads/2010/09/compare_varnish.jpg % total    % received % xferd   Average Speed   Time    Time      time  current dload  upload   total   spent     left  speed100  104k  100  104k    0     0   52793      0  0:00:02  0:00:02 --:--:--  88961[[email protected] mytest]# ls |grep part | xargs du -sh4.0k     one.part1112k    three.part34.0k    two.part2
When you use them, catimg.part* >img.jpg
12, download progress information is not displayed
[Email protected] mytest]# curl-s-o aaa.jpg http://www.codesky.net/wp-content/uploads/2010/09/compare_varnish.jpg
13, Show download progress bar
[Email protected] mytest]# Curl-#-O http://www.codesky.net/wp-content/uploads/2010/09/compare_varnish.jpg######## ################################################################ 100%
14. Download files via FTP
[[email protected] ~]$ curl-u Username: password-o http://www.codesky.net/demo/curtain/bbstudy_files/style.css% total% Received    % Xferd Average Speed time Time time current Dload Upload Total spent left Speed101 1934 101 1934 0 0 3184 0--:--:----:--:----:--:--7136 or in the following way
[[email protected] ~]$ Curl-o ftp://user name: Password @ip:port/demo/curtain/bbstudy_files/style.css
15, upload via FTP
[[email protected] ~]$ curl-t test.sql ftp://user name: Password @ip:port/demo/curtain/bbstudy_files/

This article from "Win under the learning of Redis" blog, declined reprint!
Perfect use of Curl under Linux to access Web pages