Linux wget command details

Wget is an open-source software developed in Linux, written by Hrvoje Niksic, and subsequently transplanted to various platforms including windows. It has the following functions and features:
(1) Support for resumable data transfer. This is also the biggest selling point of the year for network ant financial and flashget. Now, wget can also use this function. users who are not good at the network can rest assured;
(2) FTP and HTTP download methods are supported at the same time. Although most software can be downloaded through HTTP, FTP download is still required in some cases;
(3) Support for proxy servers. For systems with high security, generally, their systems are not directly exposed on the Internet. Therefore, support for proxy is a required function for downloading software;
(4) easy to set; maybe, users who are used to the graphic interface are not too familiar with command line. However, the command line has more advantages in setting, at least, the mouse can be clicked many times, and do not worry if the mouse is wrong;
(5) The program is small and completely free. The program is small and negligible, because the hard disk is too large now. If it is completely free, you have to consider it. Even if there are many so-called free software on the network, however, advertisements for these software are not what we like;

Although wget is powerful, it is relatively simple to use. The basic syntax is: wget [parameter list] URL. The following uses a specific example to describe how to use wget.
1. Download the entire HTTP or FTP site.
Wget http://place.your.url/here
This command can download the http://place.your.url/here home page. Using-X will force the creation of identical directories on the server. If the-nd parameter is used, all downloaded content on the server will be added to the local directory.

Wget-r http://place.your.url/here
This command downloads all directories and files on the server in a recursive way. The essence is to download the entire website. This command must be used with caution, because during the download, all the addresses pointed to by the downloaded website will be downloaded in the same way. Therefore, if this website references other websites, the referenced website will also be downloaded! For this reason, this parameter is not commonly used. You can use the-l number parameter to specify the download level. For example, to download only two layers, use-L 2.

If you want to make an image site, you can use the-M parameter, for example, wget-M http://place.your.url/here
At this time, wget will automatically determine the appropriate parameters to create an image site. Then, wgetwill be uploaded to the server and read to robots.txtand executed according to robots.txt.

2. resumable upload.
When the file size is very large or the network speed is very slow, the connection is often cut off before the file is downloaded. In this case, resumable data transfer is required. The resumable upload of wget is automatic. You only need to use the-C parameter, for example:
Http://the.url.of/incomplete/file wget-C
Resumable data transfer requires the server to support resumable data transfer. The-t parameter indicates the number of retries. For example, if you need to retry 100 times, write-t 100. If it is set to-T 0, it indicates an infinite number of retries until the connection is successful. The-t parameter indicates the timeout wait time, for example,-T 120, indicating that a timeout occurs even if the connection fails for 120 seconds.

3. Batch download.
If multiple files are downloaded, you can generate a file, write the URL of each file in a line, such as the generated file download.txt, and then run the command wget-I download.txt.
This will download all the URLs listed in download.txt. (If the column is a file, download the file. If the column is a website, download the homepage)

4. Selective download.
You can specify that wget only downloads one type of files, or does not download any files. For example:
Wget-m-reject = GIF http://target.web.site/subdirectory
Download http://target.web.site/subdirectory, but the GIF file is omitted. -Accept = list acceptable file types,-reject = List reject accepted file types.

5. Password and authentication.
Wget can only process websites restricted by user name/password. Two parameters can be used:
-Http-user = User: Set the HTTP user
-Http-passwd = pass: Set the HTTP Password
Websites that require certificate authentication can only use other download tools, such as curl.

6. Use the proxy server for download.
If your network needs to go through the proxy server, you can have wget download files through the proxy server. Create a. wgetrc file in the current user directory. You can set the proxy server in the file:
HTTP-proxy = 111.111.111.111: 8080
FTP-proxy = 111.111.111.111: 8080
Indicates the HTTP Proxy server and the FTP Proxy Server respectively. If the proxy server requires a password, use:
-Proxy-user = User: sets the proxy user.
-Proxy-passwd = pass: sets the proxy password.
These two parameters.
Use the-proxy = on/off parameter to use or disable the proxy.
Wget also has many useful functions that need to be mined by users.

Appendix:

Command Format:
Wget [parameter list] [target software and web site]

-V,-version: displays the software version number and exits;
-H and-help show the software help information;
-E,-Execute = command to execute a ". wgetrc" command

-O,-output-file = file: Save the software output information to the file;
-A,-append-output = file: append the software output information to the file;
-D and-Debug display the output information;
-Q,-Quiet does not display output information;
-I,-input-file = file: Get the URL from the file;

-T,-tries = number indicates the number of downloads (0 indicates infinite times)
-O-output-document = file: The downloaded file is saved as another file name.
-NC,-no-clobber do not overwrite existing files
-N,-timestamping only downloads new files than local
-T,-Timeout = seconds: Set the timeout time.
-Y,-proxy = On/Off disable proxy

-Nd,-no-directories do not create a directory
-X,-force-directories force Directory Creation

-Http-user = User: Set the HTTP user
-Http-passwd = pass: Set the HTTP Password
-Proxy-user = User: sets the proxy user.
-Proxy-passwd = pass: sets the proxy password.

-R,-recursive download the entire website and directory (use it with caution)
-L,-level = Number download level

-A,-accept = list acceptable file types
-R,-reject = the type of the file that the List rejects.
-D,-domains = list acceptable domain names
-Exclude-domains = List rejected domain names
-L,-relative download link
-Follow-FTP: only download the FTP Link
-H,-span-hosts can download external hosts
-I,-include-directories = List Directory
-X,-exclude-directories = List reject directory

The Chinese document name is usually encoded, but it is normal in-cut-dirs,
Wget-r-NP-NH-cut-dirs = 3 ftp: // host/test/
Test. txt
Wget-r-NP-NH-nd ftp: // host/test/
When b4%fa%b8%d5.txt
Wget "ftp: // host/test /*"
When b4%fa%b8%d5.txt

The unknown reason may be that wget automatically uses encode_string to process the part of the captured file name to avoid special file names, therefore, this patch processes the content "% 3A" by encode_string, restores it to ":" using decode_string, and applies it to the directory and file name, decode_string is the built-in function of wget.

Wget-T0-C-nh-x-NP-B-m-P/home/sunny/nod32view/http://downloads1.kaspersky-labs.com/bases/-O wget. Log