Download programs for Linux

Download programs on the Linux platform-general Linux technology-Linux programming and kernel information. The following is a detailed description. On Windows platforms, such as NetAnts, "Internet Express", and TelePro, there are also powerful offline browsers like WebZip. These tools enable us to easily download files, directories, part of the website, and even the entire website in Windows. However, in Linux, there are few such tools. In the integrated development environment KDevelop 1.2, a website download program is implemented, which supports "multi-thread download" and "resumable upload" at the file level ". Basic Principles
1. hyperlink path Finding Algorithm
To download all or part of the content of a website flexibly, the program must be able to traverse the entire website from the URL specified by the user along its contained hyperlink. Based on the user's restrictions, filter out the files to be downloaded.
From the perspective of graph theory, a website is actually a "connected directed graph" composed of files and hyperlinks ". The file is the vertex in the graph, and the hyperlink is a directed edge. We need to perform "breadth-first traversal" on this directed graph ". Therefore, a queue URLQueue is required to store the target to be accessed. Initially, the queue only contains the URL specified by the user. The program obtains the download target URL from the queue header. If the URL meets the user's requirements, it downloads the file it points. Analyze this file, find out the hyperlink, generate the URL of the new download target, and insert them to the end of the queue. Repeat the preceding process until there is no URL in the queue that meets the User restrictions.
Because the website is a "connected directed graph", it is likely to return to the accessed file along the hyperlink. To avoid an endless loop in the program, you must register the target that has been accessed. When analyzing the hyperlink of the downloaded file, we need to compare the generated new target URL with the accessed URL, removing the URL that will cause repeated access. To improve the query speed, we use a "hash table" to store the URLs retrieved from the queue header. The "hash function" can be implemented by adding the characters in the URL as integer values and modulo a prime number. In the program described in this article, the prime number is 103.
2. multi-threaded download and resumable upload
Multi‑thread download and resumable upload use the same technology. The HTTP protocol allows the client to use the Range: bytes = a1-a2 option when sending a GET request to the server to download a file, the server is required to transmit only part of the content in the specified file from the a1 byte to the a2 byte. Therefore, when downloading an object, you can divide it into several segments, start multiple threads, establish a link with the server, and transmit multiple parts of the object separately. Finally, splice it into a complete file locally. Because the bottleneck for downloading files from the Internet is in the process of server and network transmission, simultaneous downloading with multiple threads will greatly increase the download speed.
When file transmission is interrupted due to a problem, the program can save the current download progress and downloaded content of each thread as "breakpoint information" to the file. When the user downloads the same object next time, the program can download the last unfinished part based on the breakpoint information stored in the object, and then splice the entire file to complete the download. This technology is very advantageous for downloading large files when there is frequent "disconnection. Therefore, this technology is used in popular download software.
How to start a thread in Linux
1. Define a function with void * as the parameter and return void. For example, to start the download thread, You need to define the following functions:
Void * start (void * arg)
{
(CWebCopy *) arg)-> DownLoad ();
Return arg;
}
2. When you need to start a thread, you only need three statements. For example, the Code for starting a download thread is as follows:
# Include
Pthread_t tid;
Pthread_create (& tid, NULL, start, arg );
Implementation of some programs
Due to limited space, Here we only provide the flowchart of the Master Program () and the download control thread (), and provide the algorithm description of the download thread.
　
Flowchart of the master program
　
Flowchart of the download control thread
　
The following is the algorithm description of the three key functions in the download thread (assuming that all these three functions are encapsulated in the CwebCopy class ).
Int CWebCopy: DownLoad (char * host, char * path, int a1, int a2, BYTE * buf)
{
// This function downloads the content from the a1 byte to the a2 byte of the file whose path is path from the HTTP server specified by the host, establish a streaming link with port 80 (HTTP port) of the host specified by the host. A total of five attempts are made.
Bool connected = false;
Int sock; // used to store socket Descriptors
For (I = 0; I <5; I ++ ){
If (sock = Connect (host, 80) <0)
Sleep (1 );
Else {
Connected = true;
Break;
}
}
If (connected ){
// Send a GET request to the specified HTTP server to download part of the file in the specified path
Send (sock, "GET path % cHost: % s % cRange: bytes = % d-% d % c", path, 10, host, 10, a1, a2, 10 );
Int inflen, index = 0;
While (1 ){
Struct timeval TV;
TV. TV _sec = 1;
TV. TV _usec = 0;
// Check whether the socket receives data. Try 20 times at an interval of 1 second.
Int readen;
Readen = ReadEn (sock, TV, 20 );
// If no data exists in the socket within 20 seconds, it is deemed to have timed out.
If (readen <1) return-1;
// Receives data and stores it in the Information buffer zone.
Inflen = read (sock, (buf + index), a2-a1); // if the desired part of this file has been downloaded
If (inflen <= 0) break;
}
Return 0;
}
Else return-1;
}
Int CWebCopy: Send (int sock, char * fmt ,...)
{
// This function sends the specified string to the remote HTTP server pointed to by sock.
Char BUF [1024];
Va_list argptr;
// Process variable number Parameters
Va_start (argptr, fmt );
// Sort the parameters into strings and put them in the BUF.
Vsprintf (BUF, fmt, argptr );
Va_end (argptr );
// Send the string in the BUF to the remote server pointed to by sock
Return send (sock, BUF, strlen (BUF), 0 );
}
Int CWebCopy: ReadEn (int sock, struct timeval TV, int tryloop)
{
// Use the select function to check socket sock. If the socket receives data, "Read enabling" is returned. If an error other than "Internal interruption" is returned, "Read prohibited" is returned"
Fd_set rfdset;
FD_ZERO (& rfdset );
FD_SET (sock, & rfdset );
Int readen = 0;
For (int I = 0; I <tryloop; I ++ ){
Readen = select (m_sock + 1, & rfdset, NULL, NULL, & TV );
If (readen> 0 | (readen <0 & errno! = EINTR ))
Break;
Sleep (1 );
}
Return readen;
}