Network management in a hodgepodge of Linux commands

In Linux, Curl is a file transfer tool that works under the command line using URL rules, which can be said to be a powerful HTTP command-line tool. It supports file upload and download, is a comprehensive transfer tool, but by tradition, the custom called URL for download tool.

As a powerful tool, curl supports a wide range of protocols including HTTP, HTTPS,ftp, post, cookies, authentication, downloading some files from specified offsets, user agent strings, speed limits, file size, progress bar, and more. To do the web process and data retrieval automation, curl can wish a helping hand.

Syntax: # curl [option] [url]

Common parameters:

-a/--user-agent <string>              Set user agent to send to server-b/--cookie <name=string/file>    cookie string or file read location-c/-- Cookie-jar <file>                    Write the cookie into this file after the operation-c/--continue-at <offset>            Breakpoint continues-d/--dump-header <file >              writes header information to the file-e/--referer                                  source URL-f/--fail                                          does not display an HTTP error when the connection fails-o/--output                                  writes the output to the file-o/-- Remote-name                      writes the output to the file, retains the filename of the remote file-r/--range <range>                      retrieves the byte range-s/--silent mute mode from the http/1.1 or FTP server                                    . Do not output anything-t/--upload-file <file>                  upload files-u/--user <user[:p assword]>      Set the server user and password-w/--write-out [ Format]                What output is completed-x/--proxy 
Example:
1. Basic usage
# Curl Http://www.linux.com
Once executed, the www.linux.com HTML will be displayed on the screen.
Ps: Since the installation of Linux is often not installed on the desktop, also means that there is no browser, so this method is frequently used to test whether a server can reach a website
2. Save the visited Web page
2.1: Save using the redirection feature of Linux
# Curl Http://www.linux.com >> linux.html
2.2: You can use Curl's built-in option:-o (lowercase) to save Web pages
$ Curl-o linux.html http://www.linux.com
After execution, the following screen appears, displaying 100% indicating a successful save
% Total    % Received% xferd Average speed time time current                                dload  Upload  Total  spent    left  Speed100 79684    0 79684    0    0  3437k      0--:--:----:---:----:--: --7781k
2.3: You can use Curl's built-in option:-O (uppercase) to save files in a Web page
Note that the URL behind this is specific to a file, or you can't catch it.
# Curl-o Http://www.linux.com/hello.sh
3. Test page return value
# Curl-o/dev/null-s-W%{http_code} www.linux.com
Ps: In the script, it is very common to test whether the website is in normal use
4. Specify proxy server and its port
Most of the time the Internet needs to use a proxy server (such as using a proxy server to surf the Internet or because of the use of curl other people's websites to block the IP address), fortunately, curl by using the built-in option:-x to support the setup agent
# curl-x 192.168.100.100:1080 http://www.linux.com
5. Cookies
Some websites use cookies to record session information. Cookies can be easily processed by browsers such as Chrome, but it is easy to process cookies in curl by adding relevant parameters.
5.1: Save the cookie information inside the HTTP response. Built-in option:-c (lowercase)
# curl-c Cookiec.txt  http://www.linux.com
Cookie information is stored in cookiec.txt after execution.
5.2: Keep the header information inside the HTTP response. Built-in option:-D
# curl-d Cookied.txt http://www.linux.com
Cookie information is stored in cookied.txt after execution.
Note: The cookie generated by-C (lowercase) and the-D cookie are not the same.

5.3: Using Cookies
Many websites monitor your cookie information to see if you have access to their website as a rule, so we need to use stored cookie information. Built-in option:-B
# curl-b Cookiec.txt http://www.linux.com
6, imitate the browser
Some websites need to use a specific browser to access them, and some need to use certain specific versions. Curl built-in option:-a allows us to specify a browser to access the site
# curl-a "mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.0) "Http://www.linux.com
So the server side will be considered to be accessed using IE8.0.
7, forged Referer (hotlinking)
Many servers check HTTP access to Referer to control access. For example: You first visit the homepage, and then visit the home page of the mailbox, where access to the mailbox Referer address is to visit the page after the successful homepage address, if the server found on the mailbox page access to the Referer address is not the home address, it is concluded that it is a stolen
The built-in OPTION:-E in curl allows us to set Referer
# curl-e "Www.linux.com" http://mail.linux.com
This will make the server think that you have clicked a link from www.linux.com.
8. Download the file
8.1: Download the file using Curl.
#使用内置option:-O (lowercase)
# Curl-o Dodo1.jpg http:www.linux.com/dodo1.jpg
#使用内置option:-O (uppercase)
# Curl-o Http://www.linux.com/dodo1.JPG
This will save the file on the server name to the local
8.2: Cyclic download
Sometimes download pictures can be the previous part of the name is the same, the last tailbone name is not the same
# Curl-o Http://www.linux.com/dodo[1-5]. Jpg
This will save the DODO1,DODO2,DODO3,DODO4,DODO5 all.
8.3: Under load name
# Curl-o Http://www.linux.com/{hello,bb}/dodo[1-5]. Jpg
Because the downloaded hello and the file name in the BB are dodo1,dodo2,dodo3,dodo4,dodo5. So the second download will overwrite the first download, so you need to rename the file.
# Curl-o #1_ #2.jpg http://www.linux.com/{hello,bb}/dodo[1-5]. Jpg
This is hello/dodo1. JPG files will become hello_dodo1 when downloaded. JPG, other files and so on, which effectively avoids overwriting the file
8.4: Sub-block download
Sometimes the download of things will be relatively large, this time we can download sections. Using the built-in Option:-r
# Curl-r 0-100-o Dodo1_part1. JPG http://www.linux.com/dodo1.JPG# curl-r 100-200-o dodo1_part2. JPG http://www.linux.com/dodo1.JPG# curl-r 200--o dodo1_part3. JPG http://www.linux.com/dodo1.JPG# cat dodo1_part* > Dodo1. Jpg
This will allow you to view the Dodo1. JPG's content.
8.5: Download files via FTP
Curl can download files via FTP, Curl provides two kinds of syntax to download from FTP
# Curl-o-u user name: password ftp://www.linux.com/dodo1.JPG# curl-o ftp://user name: Password @www.linux.com/dodo1.jpg
8.6: Show Download progress bar
# Curl-#-O Http://www.linux.com/dodo1.JPG
8.7: Download Progress information is not displayed
# curl-s-O Http://www.linux.com/dodo1.JPG
9, the breakpoint continues to pass
In Windows, we can use software such as Thunderbolt to continue the breakpoint. Curl can also achieve the same effect with built-in option:-c
If you are downloading Dodo1. JPG in the process of a sudden drop off, you can use the following ways to continue to pass
# curl-c-O Http://www.linux.com/dodo1.JPG
10. Uploading Files
Curl can not only download files, but also upload files. With built-in option:-t to achieve
# curl-t Dodo1. Jpg-u User name: Password ftp://www.linux.com/img/
This uploads the file Dodo1 to the FTP server. Jpg
11. Display Crawl Error
# curl-f Http://www.linux.com/error
Other parameters (translated here for reprint):
-a/--append when uploading a file, attaching to the destination file--anyauth can use the "any" authentication method--basic Using the HTTP Basic authentication-b/--use-ascii using ASCII text transfer-d/--data <data> HTTP POST                          Way to transmit data--data-ascii <data> post data in ASCII--data-binary <data> post data in binary mode--negotiate Use HTTP authentication--digest use digital authentication--disable-eprt Disable the use of EPRT or Lprt--dis                  ABLE-EPSV prohibit using Epsv--egd-file <file> to set EGD socket path for random data (SSL)--tcp-nodelay Use the Tcp_nodelay option-e/--cert <cert[:p asswd]> client certificate file and password (SSL)--cert-type <type> certificate file type (DER /pem/eng) (SSL)--key <key> private key Name (SSL)--key-type <type> private key file type (Der/pem/eng) ( SSL)--pass <pass> private key password (SSL)--engine <eng> encryption engine use (SSL). "--engine list" for List--cacert <file&Gt CA certificate (SSL)--capath <directory> CA (made using C_rehash) to verify peer against (SSL)--ciphers <list > SSL password--compressed Requirements Return is a compressed situation (using deflate or gzip)--connect-timeout &LT;SECONDS&G    T              Set the maximum request time--create-dirs Create a directory hierarchy of local directories--crlf upload is to convert LF into Crlf--ftp-create-dirs If the remote directory does not exist, create a remote directory--ftp-method [MULTICWD/NOCWD/SINGLECWD] control the use of CWD--FTP-PASV use PASV/EPSV instead of ports                  --ftp-skip-pasv-ip when using PASV, ignore the IP address--ftp-ssl try to use SSL/TLS for FTP data transfer--FTP-SSL-REQD      Requires SSL/TLS for FTP data transfer-f/--form <name=content> analog HTTP form submission data-form-string <name=string>                      Simulate HTTP form submission data-g/--globoff disable URL sequences and ranges use {} and []-g/--get to send data in a get way-h/--help Help-h/--header <line> Custom header information passed to the server--ignore-content-length the length of the HTTP header information ignored-i/--include                    Include protocol header information when output-i/--head only show Document information-j/--junk-session-cookies ignore session co when reading files                  Okie--interface <interface> use specified network interface/address--KRB4 <level> Use the krb4-k/--insecure of the specified security level  Allows you to not use a certificate to SSL site-k/--config specified profile read-l/--list-only lists the file names under the FTP directory--limit-rate <rate> Set transfer speed--local-port<num> Force local port number-m/--max-time <seconds> set maximum transfer time                    --max-redirs <num> Set the maximum number of read directories--max-filesize <bytes> set the maximum amount of files to download-m/--manual                          Show full manual-n/--netrc read the user name and password from the Netrc file--netrc-optional use. netrc or URL to overwrite-N--NTLM Use HTTP NTLM authentication-n/--no-buffer disable buffered output-p/--proxytunnel use HTTP proxy--pr Oxy-anyauth Select either Proxy authentication method--proxy-basic use Basic authentication on the proxy--proxy-digest use a number on the proxy        Authentication--PROXY-NTLM            Use NTLM authentication on the agent-p/--ftp-port <address> use port address instead of Pasv-q/--quote <cmd> file transfer before                  , send a command to the server--range-file read (SSL) random files-r/--remote-time when generating files locally, keep remote file time--retry <num> When there is a problem with the transmission, the number of retries--retry-delay <seconds> transmission problems, set the retry interval--retry-max-time <seconds> Pass When a problem occurs, set maximum retry time-s/--show-error display error--SOCKS4 


Reference: Http://man.linuxde.net/curl
Http://www.linuxdiyf.com/linux/2800.html
Network management in a hodgepodge of Linux commands