Linux Learning (ii) Use of the wget command

Recently, some operations under Linux, in many places have used the wget command, to record some of the usage of wget:

wget is an open source software developed under Linux, the author is Hrvoje Niksic, and was later ported to various platforms, including Windows. It has the following features and features:
(1) supports the breakpoint down-pass function, which is also the network Ant and flashget the biggest selling point of the year, now, wget can also use this feature, those networks are not too good users can rest assured;
(2) supports both FTP and HTTP download methods Although most of the software can now be downloaded by HTTP, there are times when you still need to download the software using FTP,
(3) Support proxy servers, and for highly secure systems, you will not normally expose your system directly to the Internet, so The support agent is necessary to download the software features;
(4) easy to set up; perhaps, accustomed to GUI user is not too accustomed to command line, but the command line in the settings actually have more advantages, at least, the mouse can be a few times less, do not worry about whether the mouse is wrong;
(5) program is small, Completely free, the program is small can be considered, because the current hard disk is too big, completely free to consider, even if there are many so-called free software, but these software ads are not what we like;

Although wget is powerful, it is relatively simple to use, and the basic syntax is: wget [parameter list] URL. Here are some examples to illustrate the use of wget.
1, download the entire HTTP or FTP site.
wget http://place.your.url/here
This command to download the Http://place.your.url/here home page. Using-X forces a directory to be identical on the server, and if you use the-nd parameter, all content downloaded by the server is added to the local current directory.

Wget-r Http://place.your.url/here
This command will follow the recursive method of downloading all directories and files on the server, essentially downloading the entire site. This command must be used with caution, because at the time of download, all the addresses that the downloaded site points to are also downloaded, so if the site references other sites, the referenced sites will be downloaded as well! For this reason, this parameter is not commonly used. You can use the-l number parameter to specify the level of the download. For example, to download only two tiers, use-l 2.

If you want to create a mirror site, you can use the-m parameter, for example: Wget-m http://place.your.url/here
At this point wget will automatically determine the appropriate parameters to make the mirror site. At this point, wget will log on to the server, read into the robots.txt and follow the robots.txt rules.

2, the breakpoint continues to pass.
When the file is particularly large or the network is particularly slow, often a file has not been downloaded, the connection has been cut off, at this point, the need to continue to pass the breakpoint. Wget's breakpoint continuation is automatic and requires only the-c parameter, for example:
Wget-c Http://the.url.of/incomplete/file
Using a breakpoint to resume requires the server to support the continuation of the breakpoint. The-t parameter indicates the number of retries, such as the need to retry 100 times, then write-T 100, if set to-T 0, indicates an infinite retry until the connection succeeds. The-t parameter indicates a time-out wait, such as-t 120, which means that waiting for 120 seconds does not connect even if it times out.

3, Bulk download.
If you have more than one file to download, you can generate a file, write one line for each file URL, such as generate file Download.txt, and then use the command: Wget-i download.txt
This will download each URL listed in Download.txt. (If the column is a file to download the file, if the column is a site, then download the first page)

4, Selective download.
You can specify that you want wget to download only one type of file, or not to download it. For example:
Wget-m–reject=gif http://target.web.site/subdirectory
Indicates that the http://target.web.site/subdirectory is downloaded, but the GIF file is ignored. –accept=list can accept the file type, –reject=list rejects the accepted file type.

5, Password and authentication.
Wget can only handle websites that restrict access using username/password, with two parameters:
–http-user=user setting up an HTTP user
–http-passwd=pass Setting the HTTP password
For sites that require certificates for certification, you can only use other download tools, such as curl.

6, the use of Proxy server for download.
If the user's network needs to go through a proxy server, then you can let wget through the proxy server for file download. At this point, you need to create a. wgetrc file in the current user's directory. You can set up a proxy server in the file:
Http-proxy = 111.111.111.111:8080
Ftp-proxy = 111.111.111.111:8080
Represents the proxy server for HTTP and the proxy server for FTP, respectively. If the proxy server requires a password, use:
–proxy-user=user setting up a proxy user
–proxy-passwd=pass Setting the proxy password
These two parameters.
Use the parameter –proxy=on/off or close the agent.
Wget also has a lot of useful features that users need to dig into.

Appendix:

Command format:
wget [parameter list] [target software, Web page URL]

-v,–version Display the software version number and then exit;
-H,–HELP display software help information;
-e,–execute=command execute a ". Wgetrc" command

-o,–output-file=file Save the software output information to a file;
-a,–append-output=file Append the software output information to the file;
-d,–debug display output information;
-q,–quiet does not display output information;
-i,–input-file=file get the URL from the file;

-t,–tries=number Download count (0 = infinite)
-o–output-document=file download file to another file name
-nc,–no-clobber do not overwrite files that already exist
-n,–timestamping only download newer files than the local
-t,–timeout=seconds setting the time-out period
-y,–proxy=on/off Closing the agent

-nd,–no-directories do not create a directory
-x,–force-directories forcing a directory to be established

–http-user=user setting up an HTTP user
–http-passwd=pass Setting the HTTP password
–proxy-user=user setting up a proxy user
–proxy-passwd=pass Setting the proxy password

-r,–recursive Download the entire website, directory (use caution)
-l,–level=number Download Hierarchy

-a,–accept=list types of files that can be accepted
-r,–reject=list rejected file types
-d,–domains=list domain names that can be accepted
–exclude-domains=list denied domain name
-l,–relative Download Associated Links
–follow-ftp Download Only FTP links
-h,–span-hosts can download the outside host
-i,–include-directories=list Allowed Directories
Directories rejected by-x,–exclude-directories=list

The Chinese document name is encoded in the usual situation, but it is normal when –cut-dirs,
Wget-r-np-nh–cut-dirs=3 ftp://host/test/
Test. txt
Wget-r-np-nh-nd ftp://host/test/
%b4%fa%b8%d5.txt
wget "ftp://host/test/*"
%b4%fa%b8%d5.txt

Due to unknown reasons, may be to avoid the special file name, wget will automatically take the part of the grab file name encode_string processed, so the patch will be encode_string processed into "%3a" this thing, with decode_string to restore ":" and applies to the part of the directory and the file name, Decode_string is the wget built-in function.

Wget-t0-c-nh-x-np-b-m-p/home/sunny/nod32view/http://downloads1.kaspersky-labs.com/bases/-o wget.log

Original address: http://linux.blogbus.com/logs/46636997.html

Linux Learning (ii) Use of the wget command