Linux instructions for curl File Transfer

Linux instructions for curl File Transfer
Command: curl
In linux, curl is a File Transfer tool that uses URL rules to work under the command line. It can be said that it is a powerful http command line tool. It supports file upload and download. It is a comprehensive transmission tool. However, traditionally, URLs are called download tools.

Working principle: the curl command is implemented using the libcurl library. The libcurl library is often used in C Programs to process HTTP requests. curlpp is a C ++ encapsulation of libcurl, these items can be used for web page capturing and network monitoring development. The curl command can help solve problems encountered during the development process.

Syntax: # curl [option] [url]


Common parameters:

-A/-- user-agent <string> sets the cookie string or file read location-c/-- cookie-that the user agent sends to the server-B/-- cookie <name = string/file>- jar <file> after the operation is complete, write the cookie to this file-C/-- continue-at <offset> resumable transfer-D/-- dump-header <file> to push the header information write to this file-e/-- referer source URL-f/-- fail connection failure does not show http Error-o/-- output writes the output to this file-O/-- remote -name: writes the output to the file, retain the remote file name-r/-- range <range> to retrieve the mute mode from the HTTP/1.1 or FTP server byte range-s/-- silent. Do not output anything-T/-- upload-file <file> upload file-u/-- user <user [: password]> set the server user and password-w/-- write-out [format] After what output is complete-x/-- proxy 



Example:

1. Basic usage

# Curl http://www.doiido.com

After the execution, the html of www.doiido.com will be displayed on the screen.

Ps: Since the desktop is not installed many times during linux installation, it also means that there is no browser. Therefore, this method is often used to test whether a server can reach a website.





2. Save the accessed webpage

2.1: Use the linux redirection function to save

# Curl http://www.doiido.com> doiido.html



2.2: You can use the built-in option:-o (lower case) of curl to save the webpage

$ Curl-o doiido.html http://www.doiido.com

After the execution is complete, the following page is displayed. If 100% is displayed, the storage is successful.


  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100 79684    0 79684    0     0  3437k      0 --:--:-- --:--:-- --:--:-- 7781k

2.3: You can use the built-in option:-O (uppercase) of curl to save files on the webpage.

Note that the url next to this file must be specific to a file. Otherwise, the file cannot be captured.

# Curl-O http://www.doiido.com/hello.sh





3. Test the webpage Return Value

# Curl-o/dev/null-s-w % {http_code} www.doiido.com

Ps: In scripts, it is common to test whether a website is used normally.





4. Specify the proxy server and its port

In many cases, a proxy server is used to access the Internet (for example, when the proxy server is used to access the Internet or the IP address is blocked by others because of the curl website). Fortunately, curl uses the built-in option: -x to support proxy settings

# Curl-x 192.168.100.100: 1080 http://www.doiido.com





5. cookie

Some websites use cookies to record session information. For browsers such as chrome, cookie information can be easily processed, but it is easy to process cookies by adding relevant parameters to curl.

5.1: Save the cookie information in the http response. Built-in option:-c (lower case)

# Curl-c cookiec.txt http://www.doiido.com

After the execution, cookieinformation is saved to cookiec.txt.

5.2: Save the header information in the http response. Built-in option:-D

# Curl-D cookied.txt http://www.doiido.com

After the execution, cookieinformation is saved to cookied.txt.



Note: The cookie generated by-c (lower case) is different from the cookie in-D.



5.3: use cookie

Many websites monitor your cookie information to determine whether you access their websites according to rules. Therefore, we need to use the saved cookie information. Built-in option:-B

# Curl-B cookiec.txt http://www.doiido.com





6. Imitate the browser

Some websites need to use specific browsers to access them, and some also need to use certain versions. Curl built-in option:-A allows us to specify A browser to access the website

# Curl-A "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.0)" http://www.doiido.com

In this way, the server will consider it to be accessed using IE8.0.





7. Counterfeit referer (leeching)

Many servers check the http access referer to control the access.

For example, you access the home page first, and then the mailbox page on the home page. the referer address for accessing the mailbox here is the address of the page after successfully accessing the home page, if the server finds that the referer address accessed by the mailbox page is not the address of the home page, it is determined that it is a stolen connection.

The built-in option in curl:-e allows us to set referer

# Curl-e "www.doiido.com" http://mail.doiido.com

In this way, the server will assume that you have clicked a link from www.doiido.com.





8. download files

8.1: Use curl to download an object.

# Use built-in option:-o (lower case)

# Curl-o dodo1.jpg http: www.doiido.com/dodo1.JPG



# Use built-in option:-O (uppercase)

# Curl-O http://www.doiido.com/dodo1.JPG

In this way, the file will be saved to the local device by name on the server.





8.2: loop download

Sometimes, you can download images with the same names as the previous ones, but the names of the final tailspine are different.

# Curl-O http://www.doiido.com/dodo?1-5=.jpg

In this way, all dodo1, dodo2, dodo3, dodo4, and dodo5 will be saved.





8.3: Download and rename

# Curl-O http://www.doiido.com/?hello,bb#/dodo?1-5=.jpg

The downloaded hello and bb files are named dodo1, dodo2, dodo3, dodo4, and dodo5. Therefore, the second download will overwrite the first download, so you need to rename the file.

# Curl-o #1 _ # 2.JPG http://www.doiido.com/?hello,bb#/dodo?1-5=.jpg

In this way, the download of the hello/dodo1.JPG file will become hello_dodo1.JPG, and other files will be pushed accordingly, effectively avoiding file overwriting.





8.4: multipart download

Sometimes the downloaded content is relatively large, so we can download it in multiple parts. Use built-in option:-r

# Curl-r 0-100-o dodow.part1.jpg http://www.doiido.com/dodo1.JPG

# Curl-r 100-200-o dodow.part2.jpg http://www.doiido.com/dodo1.JPG

# Curl-r 200--o dodow.part3.jpg http://www.doiido.com/dodo1.JPG

# Cat dodow.part *> dodo1.JPG

In this way, you can view the dodo1.JPG content.





8.5: download files through ftp

Curl can be used to download files through ftp. curl provides two types of syntax for downloading files from ftp

# Curl-O-u User name: Password ftp://www.doiido.com/dodo1.JPG

# Curl-O ftp: // User name: password @ www.doiido.com/dodo1.JPG





8.6: display the download progress bar

# Curl-#-O http://www.doiido.com/dodo1.JPG





8.7: the download progress is not displayed.

# Curl-s-O http://www.doiido.com/dodo1.JPG





9. resumable upload

In windows, we can use software such as thunder for resumable data transfer. Curl can achieve the same effect through built-in option:-C.

If the download process of dodo1.JPG suddenly drops, you can resume the download in the following ways:

# Curl-C-O http://www.doiido.com/dodo1.JPG





10. upload files

Curl can not only download files, but also upload files. Implemented through built-in option:-T

# Curl-T dododo1.jpg-u Username: Password ftp://www.doiido.com/img/

In this way, the file dodo1.JPG is uploaded to the ftp server.





11. Capture Error

# Curl-f http://www.doiido.com/error





12. View response header information

# Curl-l http://www.doiido.com

Other parameters:


-A/-- append: When uploading a file, attach to target file -- anyauth can use "any" authentication method -- basic use HTTP basic Authentication-B/-- use-ascii use ASCII text transmission-d/-- data <data> HTTP POST data Transmission -- data-ascii <data> post data in ascii mode -- data-binary <data> post data in binary mode -- negotiate uses HTTP Authentication -- digest uses numbers authentication -- disable-eprt disable the use of EPRT or LPRT -- disable-epsv disable the use of EPSV -- egd-file <file> as random data (SSL) set EGD socket path -- tcp-nodelay use TCP_NODELAY option-E/-- cert <cert [: passwd]> client certificate file and password (SSL) -- ce Rt-type <type> Certificate file type (DER/PEM/ENG) (SSL) -- key <key> private key file name (SSL) -- key-type <type> private key file type (DER/PEM/ENG) (SSL) -- pass <pass> private key password (SSL) -- engine <eng> encryption engine usage (SSL ). "-- engine list" for list -- cacert <file> CA certificate (SSL) -- capath <directory> CA directory (made using c_rehash) to verify peer against (SSL) -- ciphers <list> SSL password -- compressed requires that the returned data be compressed (using deflate or gzip) -- connect-timeout <seconds> sets the maximum request time -- create -Dirs: create the directory hierarchy of the local directory -- crlf upload converts LF into CRLF -- ftp-create-dirs. If the remote directory does not exist, create a remote directory -- ftp-method [multicwd/nocwd/singlecwd] to control the use of CWD -- ftp-pasv uses PASV/EPSV instead of port -- When ftp-skip-pasv-ip uses PASV, ignore this IP address -- ftp-ssl tries to use SSL/TLS for ftp data transmission -- ftp-ssl-reqd requires the use of SSL/TLS for ftp data transmission-F/-- form <name = content> simulate http form submission data-form-string <name = string> simulate http form submission data-g/-- globoff disable URL sequence and range use {} and []-G /-- get sends data in get mode-h/-- help-H/-- hea Der <line> Custom header information is transmitted to the server. The length of HTTP header information ignored by ignore-content-length-I/-- include contains the protocol header information-I/-- head only show document information-j/-- junk-session-cookies when reading files ignore session cookies -- interface <interface> use the specified network interface/address -- krb4 <level> use the specified security level krb4-k/-- insecure allows reading-l/-- list-only to list file names under the ftp directory without using certificates to the SSL site-K/-- config rate> set transmission speed -- local-port <NUM> force use the local port number-m/-- max-time <seconds> to set the maximum transmission time-max-redirs <num> Max-filesize <bytes> specifies the maximum number of downloaded files-M/-- manual displays all manual-n/-- netrc reads the user name and password from the netrc file-netrc -Use optional. netrc or URL to override-n -- ntlm use http ntlm authentication-N/-- no-buffer disable buffer output-p/-- proxytunnel use HTTP proxy -- proxy-anyauth select any proxy identity verification Method -- proxy-basic use basic authentication on the proxy -- proxy-digest use digital authentication on the proxy -- proxy-ntlm use ntlm authentication on the proxy-P/-- ftp -port <address> use the port address, instead of sending the command to the server -- range-file to read (SSL) random file-R before using PASV-Q/-- quote <cmd> file Transfer /-- Remote-time when a file is generated locally, retain the remote file time -- retry <num> the number of retries -- retry-delay <seconds>, set the retry Interval -- retry-max-time <seconds> when transmission fails, set the maximum retry time-S/-- show-error to display the error -- socks4