Download HTTPS link for wget

Use the following command:

Wget-r-NP-Nd -- accept = GZ -- no-check-certificate https://www.xxx.com/dir/ -- http-user = username -- http-Password = Password

To download all. GZ file under "dir" Directory

-NP means no parent directory
-Nd means don't construct local directory structure
-- Accept = GZ means only download. GZ files in the directory

 

HTTPS (SSL/TLS) Options(HTTPS (SSL)Parameter options)
-- Certificate = File	Optional client segment Certificate
-- Private-Key = File	"Key document" available for this certificate"
-- Private-key-type = Type	The "Key type" available for this certificate
-- EGD-file = File	EGD socket document name
-- Ca-directory = directory	CA hash List Directory
-- Ca-certificate = File	Documents containing Ca
-- Certificate-type = [PEM (default), der]	Client-Cert Type: PEM, der
-- No-check-Certificate	Do not check the server certificate
-- Secure-Protocol = [auto, SSLv2, SSLv3, tlsv1 ]	Select SSL protocols: auto, SSLv2, SSLv3, tlsv1
FTP options(FTPParameter options)
-- FTP-user	User name used to log on to FTP (Note: The best way is to define it in the. netrc or. wgetrc file)
-- FTP-Password	Login FTP password (Note: The best way is to define it in the. netrc or. wgetrc file)
-- No-Remove-listing	Do not delete the ". Listing" Document
-- No-glob	Disable ftp document names for all wildcard characters
-- No-passive-FTP	Disable "passive" Transmission Mode
-- Retr-symlinks	In recursive mode, download the document indicated by the Link (excluding the connected directory)

 

 1. Download a single file
Wget URL + filename

Four items are displayed during the download process.
Download Ratio
Downloaded size
Current download speed
Remaining Time

2. Use an uppercase o as the parameter to save it
Wget-O save_name URL

This method is applicable when no explicit file name exists in the corresponding link.

Example: wget-O xx.zip http://www.vim.org/scripts/download_script.php? Src_id = 1, 7701

Download it again without the-O parameter.

Ls-Al
Total 132
Drwxr-XR-x 2 root Root 4096 07-12.
Drwxr-XR-x 4 Root 4096 07-11 16: 26 ..
-RW-r -- 1 Root 50243 07-12 10:43 download_script.php? Src_id = 1, 7701
-RW-r -- 1 Root 50243 07-12 10:43 xx.zip

We found that the download size is the same. But the file name must be converted once without the-O parameter. It is better to use the-O parameter.

Mv "download_script.php? Src_id = 7701 "yy.zip

3. Specify the download speed
The method is to use wget -- limit-Rate

The wget program uses all the bandwidth by default. If
It is unacceptable to download a large file on the production server.
To avoid this, use the -- limit-rate parameter.
Wget -- limit-rate = 200 K http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

4. breakpoint download

Use wget-C to complete unfinished downloads

To download the package, you need to stop and do other things. You can pause with ^ C.

After you return, you can add the-C parameter to continue downloading.

NOTE: If-C is not added, the downloaded file will have an additional. 1 suffix.

5. Download in the background
Method: Add a-B parameter.

Wget-B URL/filename
For background download. The download is written to the wget-log file.

Use tail-F wget-log to view the download log

6. Simulate download in a browser

Some websites cannot be downloaded in non-browser environments. Use -- User-Agent to set

Wget -- User-Agent = "Mozilla/5.0 (X11; U; Linux i686; en-US; RV: 1.9.0.3) Gecko/2008092416 Firefox/3.0.3" URL-TO-DOWNLOAD

7. Test download link
Method: Use -- spider

When trying to schedule a download, you must first check whether the download link is valid.

Wget -- spider DOWNLOAD-URL

If OK is returned, the download link is correct!

For example
Wget -- spider "http://ip138.com/ips.asp? IP = 58.251.193.133 & Action = 2"
Spider Mode Enabled. Check if remote file exists.
-- 11:36:32 -- http://ip138.com/ips.asp? IP = 58.251.193.timeout & Action = 2
Parsing host ip138.com... 221.5.47.136
Connecting to ip138.com | 221.5.47.136 |: 80... connected.
An HTTP request has been sent and is waiting for response... 200 OK
Length: 7817 (7.6 K) [text/html]
Remote file exists and cocould contain further links,
But recursion is disabled -- not retrieving.

8. Increase the number of attempts
Method: -- tries = 1000.
If the network speed is incorrect, an error may occur when downloading large files,
By default, wget tries 20 links.

If you try 75 times, you can
Wget -- tires = 75 DOWNLOAD-URL

9. Download multiple files and use wget-I
Write multiple download links to a download-file-list.txt file, and then use

Wget-I download-file-list.txt

10. Download the entire site
Method: Use the -- mirror parameter.

When you want to download a complete site and perform local browsing,
Wget -- mirror-p -- convert-links-P./LOCAL-DIR WEBSITE-URL

Parameter description:
-- Mirror: set this parameter to create a local image.
-P: Download all elements suitable for displaying HTML files.
-- Convert-links
-P./LOCAL-DIR: Save all files and directories to the specified folder

11. Downloading specified types of files is prohibited.

For example, when downloading a website, you do not want to download GIF animated images.

Wget -- reject = GIF WEBSITE-TO-BE-DOWNLOADED.

12. Record download logs
Method: Use lowercase letter O

Wget-O xx.html. Log-O xx.html "http://ip138.com/ips.asp? IP = 58.251.193.133 & Action = 2"

Check the log:
[Root @ localhost opt] # Cat xx.html. Log
-- 11:57:22 -- http://ip138.com/ips.asp? IP = 58.251.193.timeout & Action = 2
Parsing host ip138.com... 221.5.47.136
Connecting to ip138.com | 221.5.47.136 |: 80... connected.
An HTTP request has been sent and is waiting for response... 200 OK
Length: 7817 (7.6 K) [text/html]
Saving to: 'xx.html'

0 k... 100% 65.5 0.1 K = s

11:57:22 (65.5 kb/s)-'xx.html 'saved [7817/7817]

13. It is an enhanced version of article 9th. Limit the download capacity

Wget-q5m-I FILE-WHICH-HAS-URLS

When the downloaded file reaches 5 MB, the download is stopped.
Note: If the list of Download Links for a single file is not,
This restriction does not take effect.

14. Opposite to 11th,
This article describes how to download only files of the specified type.

Download all PDF files from a website

Wget-r -a.pdf http: // URL-to-webpage-with-pdfs/

15. Use wget to download FTP

Anonymous FTP download is similar to HTTP download
Wget FTP-URL.

If you need to enter the user name and password

Wget -- FTP-user = username -- FTP-Password = PASSWORD DOWNLOAD-URL