Wget User Guide

Wget is a free tool for automatically downloading files from the network. It supports HTTP, https, and FTP protocols and can use HTTP proxy.

The so-called automatic download means that wget can be executed in the background after the user exits the system. This means that you can log on to the system, start a wget download task, and exit the system. wget will be executed in the background until the task is completed, compared with most other browsers, users need to participate in downloading a large amount of data, which saves a lot of trouble.

Wget allows you to track links on the HTML page and download them to create the local version of the remote server, completely recreating the directory structure of the original site. This is often called recursive download ". During recursive download, wget complies with the robot exclusion standard (/robots.txt). wget can convert the link to a local file while downloading to facilitate offline browsing.

Wget is very stable. It has strong adaptability to unstable networks with narrow bandwidth. if the download fails due to network reasons, wget will continue to try until the entire file is downloaded. If the server interrupts the download process, it will be again connected to the server to continue the download from the stopped place. This is very useful for downloading large files from servers with limited connection time.

Common wget usage

Wget format

Usage: wget [OPTION]... [URL]...

• Use wget for site image:

Wget-r-p-NP-K http://dsec.pku.edu.cn /~ Usr_name/# Or wget-M http://www.tldp.org/LDP/abs/html/

• Download a part of the downloaded files from an unstable network and during free time.

Wget-T 0-W 31-C http://dsec.pku.edu.cn/BBC.avi-O down. log & # or read the list of files to be downloaded from filelist wget-T 0-W 31-C-B ftp://dsec.pku.edu.cn/linuxsoft-I filelist.txt-O down. log &

The above code can also be used for downloading when the network is relatively idle. My usage is: In mozillawill not easily download the urlchain to the memory, then paste it to the filelist.txt file, and execute the second code above before going out of the system at night.

• Download via proxy

wget -Y on -p -k https://sourceforge.net/projects/wvware/

The proxy can be set in the environment variable or wgetrc file.

# Set proxy export proxy = http: // 211.90.168.94: 8080/# In ~ /. Wgetrc set proxy http_proxy = http://proxy.yoyodyne.com: 18023/ftp_proxy = http://proxy.yoyodyne.com: 18023/

Wget option category list

• Start

-V, -- version: After wget is displayed, exit-h, -- help print syntax help-B, -- background is started and transferred to the background for execution-E, -- execute = Command Execution '. wgetrc 'command. For the wgetrc format, see/etc/wgetrc or ~ /. Wgetrc

• Record and input file

-O, -- output-file = file: Write the record to the file-a, -- append-output = file: append the record to the file-D, -- debug print debug output-Q, -- Quiet quiet mode (no output)-V, -- verbose lengthy mode (this is the default setting)-NV, -- Non-verbose turn off the lengthy mode, but it is not quiet mode-I, -- input-file = file the URLs-F that appears in the file, -- force-HTML treats the input file as an HTML file-B, -- base = URL uses the URL as the prefix of the relative link in the file specified by the-f-I parameter -- sslcertfile = file optional client certificate -- sslcertkey = Keyfile optional client certificate Keyfile -- EGD-file = file specifies the file name of the EGD socket.

• Download

-- Bind-address = address specifies the local address (host name or IP address, used when there are multiple local IP addresses or names)-t, -- tries = Number sets the maximum number of attempts (0 indicates no limit ). -o -- output-document = file: Write the document to the file-nc. -- no-clobber should not overwrite the existing file or use it. # prefix-C, -- continue, and then download the undownloaded file -- Progress = type to set the process flag-N, -- timestamping do not re-download the file except for the new-s, -- server-response: prints the response from the server -- Spider does not download anything-T, -- timeout = seconds sets the number of seconds for response timeout-W, -- Wait = seconds two attempts interval seconds -- waitretry = seconds waits for 1... seconds seconds
-- Random-Wait waits for 0 seconds between downloads... 2 * Wait
-Y, -- proxy = On/Off enable or disable proxy-Q, -- quota = Number sets the download capacity limit -- limit-rate = rate to limit the download rate

• Directory

-Nd -- no-directories: do not create directory-X, -- force-directories force create directory-NH, -- no-host-directories do not create host directory-P, -- directory-Prefix = Prefix: save the file to the directory prefix /... -- Cut-dirs = Number ignore the remote directory of the number layer

• HTTP OPTIONS

-- Http-user = User: Set the HTTP user name to user. -- http-passwd = pass sets the HTTP password to PASS.-C, -- cache = ON/OFF allows/does not allow server-side data caching (usually ). -E, -- HTML-Extension: Save all text/html files with the. html extension -- ignore-length ignore 'content-length' header field -- header = string insert string in headers -- proxy-user = user settings the proxy username is user -- proxy-passwd = pass. Set the proxy password to pass -- Referer = URL. The HTTP request contains 'Referer: URL 'header-S, -- save-headers Save the HTTP header to the file-u, -- User-Agent = agent sets the proxy name as agent rather than wget/version. -- no-http-keep-alive disable the HTTP activity Link (permanent link ). -- cookies = off do not use cookies. -- load-Cookies = File Load the cookie from the file before starting the session -- save-Cookies = File Save the cookies to the file after the session ends

• FTP options

-Nr, -- Dont-Remove-listing does not remove '. listing 'file-G, -- glob = on/off open or close the globbing mechanism of the file name -- passive-FTP use passive transmission mode (default value ). -- active-FTP uses the active transmission mode -- retr-symlinks recursively points the link to a file instead of a directory)

• Recursive download

-R, -- Recursive recursive download-use with caution! -L, -- level = maximum recursive depth of number (inf or 0 indicates infinity ). -- delete-after the current time, partial deletion of the file-K, -- convert-links convert non-relative link to relative link-K, -- backup-converted before converting file X, back up X. orig-M, -- mirror is equivalent to-r-N-l INF-Nr. -P, -- page-requisites download and display all images of HTML files

• Include and exclude (accept/reject) in recursive download)

-A, -- accept = List the list of accepted extensions separated by semicolons-R, -- reject = List the list of unaccepted extensions separated by semicolons-D, -- domains = List the list of accepted domains separated by semicolons -- exclude-domains = List the list of untrusted domains separated by semicolons -- follow-FTP tracking FTP links in HTML documents -- follow- tags = List the list of Tracked HTML tags separated by semicolons-G, -- ignore-tags = List the list of ignored HTML tags separated by semicolons-h, -- span-hosts is recursively transferred to the external host-l, -- relative only traces relative links-I, -- include-directories = List list of allowed directories-X, -- exclude-directories = List list of excluded directories-NP, -- no-parent should not be traced back to the parent directory