Curl and wget command

1. Curl

Curl Support HTTP, HTTPS, FTP and other protocols, but also support POST, cookies, authentication, from the specified offset to download some files, user-agent, speed limit, file size, progress bar features. 1.1 Options

-	-
-A or--user-agent <string>	Setting up user agents
-B or--cookie <name=string/file>	Cookie string or file read location
-C or--cookie-jar <file>	Write cookies to file after operation ends
--basic	Using HTTP Basic Authentication
-D or--data <data>	POST Mode transfer data
--data-ascii <data>	POST data in ASCII mode
--data-binary <data>	POST data in binary mode
--tcp-nodelay	Using the Tcp_nodelay option
-E or--referer	Source URL
-E or--cert <cert[:p asswd]>	Client certificate files and passwords (SSL)
--cert-type <type>	Certificate file type (Der/pem/eng) (SSL)
--key <key>	Private key file name (SSL)
--key-type <type>	Private key file Type (Der/pem/eng) (SSL)
--pass <pass>	Private key Password (SSL)
--engine <eng>	Encryption engine using (SSL)
--cacert <file>	CA Certificate (SSL)
--capath <directory>	CA Path
--ciphers <list>	SSL Password
-F or--form <name=content>	Simulate HTTP form submission data
--form-string <name=string>	Simulate HTTP form submission data
-H or--header <line>	Custom header information is passed to the server
-I or--head	Show Request header information only
-T or--upload-file <file>	Uploading files
-O or--output	Write the output to the file
-O or--remote-name	Write output to file, keep file name of remote file
-S or--silent	Silent mode. Don't output anything.

1.2 Example look at the returned HTTP Header only

[Root@vm_139_74_centos ~]# curl-i www.163.com
http/1.1 OK
Expires:sun, may 2018 15:22:03 GMT Date:su
N, may 2018 15:20:43 GMT
server:nginx
content-type:text/html; CHARSET=GBK
transfer-encoding:chunked< C6/>vary:accept-encoding,user-agent,accept
cache-control:max-age=80
age:44
x-via:1.1 PSfjqzdxgn15:0 (CDN cache Server V2.0), 1.1 chongdianxin121:4 (CDN cache server V2.0)
connection:keep-alive
x-dscp-value:0

File Download

Curl default behavior is to download, browse the page is to download the Web page to the local, Curl URL will be the corresponding file download the URL. However, curl will output the downloaded file to the STDOUT window, which can be written to the file through the-O or-o option.

[Root@vm_139_74_centos ~]# Curl http://img02.tooopen.com/images/20160509/tooopen_sy_161967094653.jpg-o 1.jpg
  % Total    % Received% xferd  Average Speed time Time current
                                 dload  Upload   Total   spent    left  Speed  208k  208k    0     0   447k      0--:- -:----:--:----:--:--  448k

Set Cookies

Use-B to read cookie files, or--cookie <name=string/file> options to specify cookies, multiple cookies separated by semicolons:

Curl http://www.baidu.com--cookie "user=root;pass=123456"

Save Cookies

Use the-C or--cookie-jar <file> options:

[Root@vm_139_74_centos ~]# Curl www.baidu.com--cookie-jar Cookie
<! DOCTYPE html> ...
[Root@vm_139_74_centos ~]# Cat Cookie
# Netscape HTTP Cookie File
# http://curl.haxx.se/docs/http-cookies.html
# This file is generated by libcurl! Edit at your own risk ...

Baidu.com  TRUE    /   FALSE   1525707455  bdorz   27315

Set HTTP Header

Use the-H or--header <line> to pass multiple header information, for example:

Curl-h "Host:www.baidu.com"-H "accept-language:en" www.baidu.com

2. wget

wget is used to download files to support breakpoint continuation. 2.1 Options

-a< log file: Records the execution process in the specified log file;
-a< suffix: Specifies the suffix name to download the file, separating multiple suffix names with commas;
-B: Running in the background wget;
-B < connection address: Set the base address of the reference connection address;
-C: Continue with the last terminal task;
-c< flag: Set the server block feature flag on to active, off to off, default to on;
-d< domain Name list ": Set down the list of domain names, the domain name between the", "separated;
-e< instruction: Executes the specified instruction as part of the file". Wgetrc ";
-i<: Gets the URL address to download from the specified file;
-L < directory list: Set down the directory list, multiple directories with "," delimited;
-Q: Do not show instruction execution
-O: Save as a file with the specified name
-r: Recursive download 2.2 Example Download File

[Root@vm_139_74_centos ~]# wget http://img02.tooopen.com/images/20160509/tooopen_sy_161967094653.jpg
-- 2018-05-06 23:42:14--  http://img02.tooopen.com/images/20160509/tooopen_sy_161967094653.jpg
Resolving Img02.tooopen.com (img02.tooopen.com) ... 61.240.138.246, 221.194.130.173
connecting to Img02.tooopen.com (img02.tooopen.com) |61.240.138.246|:80 ... Connected.
HTTP request sent, awaiting response ... OK
length:213639 (209K) [image/jpeg]
saving to: ' Tooopen_sy_161967094653.jpg.1 '

100%[============== =============================================================================================================== ========================>] 213,639     --.-k/s in   0.09s   

2018-05-06 23:42:14 (2.17 MB/s)-' Tooopen_sy_ 161967094653.jpg.1 ' saved [213639/213639]

downloading files and renaming

[Root@vm_139_74_centos ~]# wget Http://img02.tooopen.com/images/20160509/tooopen_sy_161967094653.jpg-O this_is_my_ Pic

download the specified format file

Wget-r-a.pdf URL

You can download all the pictures, videos, and PDF files of a Web site recursively (-r option).