LINUX MAC axel--A more powerful multi-threaded download tool than wget

Objective

Recently use wget download Baidu cloud resources, slow, in a friend recommended multi-threaded download found wget is actually just a single-threaded download tool, in the face of the file will appear very chicken, and there are many criticisms, such as non-breakpoint re-connect and so on.

Axel Introduction

After some search found that there is a very useful download tool named Axel, and Wget is the same as the command line download tool, but support multi-threaded download, breakpoint re-connect and other powerful features.

The following is an introduction to the English language in man and its translation:

Axel- is from a FTP or HTTP server through multiple connection. Each connection downloads it own part of the file. Unlike most other programs, Axel downloads all of the data directly to the destination file.　　　　It saves some time at the end of because the program does not having to concatenate all the downloaded parts. Axel supports HTTP, HTTPS, FTP and FTPS protocols.

Axel- powerful command line Downloader Axel is a multi-threaded downloader that downloads files from an FTP or HTTP server. Each thread downloads a portion of the file. Unlike other downloads, Axel can download some data directly from the corresponding resource, which can save a lot of time by downloading resources instead of the overall download.

Common parameters Introduction

　　-H

That is--help, which is the most useful command, haha

　　-N X

That is--num-connections=x, the number of download threads

　　-Q

That is--quiet, no output will be output to the standard output stdout

　　-A

That is--alternate, will show an alternate program download status bar, you can see the download of different threads

　　-U x

That is--user-agent=x, the user agent that sets the request header. Some servers differentiate this parameter. The default User agent is Alex

　　

　　-T X

That is--timeout=x, set the upper limit for I/O and connection time

　　-V

That is--version, get Axel version information

　　

　　　　

Experience Summary

Here is a list of my favorite commands to use, with a strong practicality

' mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) applewebkit/605.1.15 (khtml, like Gecko) version/11.1 safari/605.1.15" c4> Directory/target filename ' destination URL

　　

Two details

1. User Agent Issues

Some websites need to emulate the user-agent of the request header to parse the URL.

So where does the request head get it? It's hard for us to simulate a user-agent request header that can be solved. User-agent can directly copy the browser--developer mode (F12 or cmd + OPTION + i), such as network---Request header, user-agent such as mine is mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) applewebkit/605.1.15 (khtml, like Gecko) version/11.1 safari/605.1.15.

2. Directory Issues

The specified directory needs to be a directory that already exists, and the command-line program usually does not create a new directory.

Suspend the downloader and run in the background

What if you want to download in the background and you want to be able to download it after you close the terminal?

Of course the use of Nohup!

Common commands:

Nohup command >outfile.  out 2>&1

The command is interpreted by suspending the command command to the background and controlling the nohup, the output of its program is redirected to the current directory Outfile.out file, and the standard error stderr is bound to stdout so that the error message is also recorded

　　

When you need to transfer the NOHUP background running program to the foreground, the jobs command to view the background job number, with the FG%n command can be recalled to the foreground run (n is the job number).

PS: A separate & can let the command background run, knowledge if the SSH session is broken,& command will automatically exit.

e.g multi-threaded download a paper and ensure that it is done in the background, even if you exit the terminal does not matter

The code is as follows:

5 ' mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) applewebkit/605.1.15 (khtml, like Gecko) version/11.1 safari/605.1.15" C5>test.pdf' http://www.aclweb.org/anthology/W03-0429 >outfile.out 2>&1 & 

　　

LINUX MAC axel--A more powerful multi-threaded download tool than wget