How to implement multi-thread resumable data transfer point-to-point

In today's network applications, file transfer is an important feature and the basis for sharing. Important protocols such as HTTP and FTP support file transfer. FTP, in particular, is the "File Transfer Protocol". Engineers designed this Protocol to solve the problem of file transfer between networks and ensure its stability and speed, simple and always maintain great vitality. As a programmer, using these existing protocols to transfer files is quite simple, but they only apply to server mode. In this way, when we want to transfer files between points, it will not work or be quite troublesome. There is a means of small use. I have always wanted to find a simple and effective method with multiple threads for resumable data transfer between vertices. After a lot of Data Reading and testing, I finally realized it, share it with you.
(I have written a utility based on this (network transfer, including source code). It can be used on an IP/TCP-based computer for your learning. :
Http://h2osky.126.com)
The implementation method (VC ++, based on TCP/IP protocol) is as follows:
Still using server and customer models, we need to design and program them separately.
The server side is relatively simple. It is mainly used to upload files, listen to customers, and transfer files. The resumable upload functions and file management are all stored on the client.
First, introduce the server:
At the beginning, we need to define a simple protocol, that is, a language that the server and client can understand. In order to simplify the problem, I only need to let the server understand two sentences. One is that the customer says "I want to read file information", and the other is "I'm ready, you can upload files ".
To implement multithreading, the function must be independent and packaged into a thread. First, a listening thread is created to access the customer and start another customer thread. I use VC ++ to implement the following:

　

DWORD winapi listenthread (lpvoid lpparam)
{

File: // socket sent from the main function
Socket pthis = (socket) lpparam;
File: // start listening
Int rc = listen (pthis, 30 );
File: // if an error occurs, the information is displayed.
If (RC <0 ){
Cstring AAA;
AAA = "Listen error/N ";
Afxgetmainwnd ()-> sendmessagetodescendants (wm_age1, (lparam) AAA. getbuffer (0), 1 );
AAA. releasebuffer ();
Return 0;
}
File: // enter the loop and receive the incoming socket
While (1 ){
File: // create a socket for client http://h2osky.126.com
Socket S1;
S1 = accept (pthis, null, null );
 
File: // send a message to the main function.
Cstring AA;
AA = "one person join! /N ";
Afxgetmainwnd ()-> sendmessagetodescendants (wm_age1, (lparam) AA. getbuffer (0), 1 );
AA. releasebuffer ();
DWORD dwthread;
File: // create a user thread
: Createthread (null, 0, clientthread, (lpvoid) S1, 0, & dwthread );
}
Return 0;

}
Next let's look at the user thread:
First look at the file message class definition:
Struct fileinfo
{
Int fileno; // file number
Int type; // what the client wants to say (the first two sentences are expressed as 1 and 2)
Long Len; // file length
Int seek; // the start position of the file, used for Multithreading

Char name [100]; // file name http://h2osky.126.com
};
File: // user thread function:
DWORD winapi clientthread (lpvoid lpparam)
{
File: // file message
Fileinfo * fiinfo;
File: // receive Cache
Char * m_buf;
M_buf = new char [100];
File: // user socket from the listener Function
Socket pthis = (socket) lpparam;
File: // read information
Int AA = readn (pthis, m_buf, 100 );
File: // if there is an error, the system returns
If (AA <0 ){
Closesocket (pthis );
Return-1;
}
File: // convert the received information into the defined file information
Fiinfo = (fileinfo *) m_buf;
　

Cstring AAA;
File: // check what the customer wants to say
Switch (fiinfo-> type)
{
File: // I want to read the file information
Case 0:
File: // read the file
AA = sendn (pthis, (char *) zmfile, 1080 );
File: // Error
If (AA <0 ){
Closesocket (pthis );
Return-1;
}
File: // send a message to the main function
AAA = "receive LIST command/N ";
Afxgetmainwnd ()-> sendmessagetodescendants (wm_age1, (lparam) AAA. getbuffer (0), 1 );
Break;
File: // I'm ready. You can upload the file.

Case 2:
File: // send a file message to the main function
AAA. Format ("% s file requested! % S/n ", zmfile [fiinfo-> fileno]. Name, nameph [fiinfo-> fileno]);
Afxgetmainwnd ()-> sendmessagetodescendants (wm_age1, (lparam) AAA. getbuffer (0), 1 );
File: // read and transmit the file
Readfile (pthis, fiinfo-> seek, fiinfo-> Len, fiinfo-> fileno );
File: // do not understand what you are talking about
Default:
AAA = "receiving protocol error! /N ";
Afxgetmainwnd ()-> sendmessagetodescendants (wm_age1, (lparam) AAA. getbuffer (0), 1 );
Break;
}

Return 0;
}

////////// Http://h2osky.126.com
File: // Read File Function
Void readfile (socket so, int seek, int Len, int Fino)
　
{
　
File: // file name
Cstring myname;

Myname. Format ("% s", nameph [Fino]);

Cfile myfile;
　
File: // open the file
Myfile. Open (myname, cfile: moderead | cfile: typebinary | cfile: sharedenynone );
File: // upload to the specified location
Myfile. Seek (seek, cfile: Begin );
Char m_buf [size];
Int len2;
Int len1;
Len1 = Len;
File: // start receiving until the entire file is sent
While (len1> 0 ){
Len2 = Len> size? Size: Len;
Myfile. Read (m_buf, len2 );
Int AA = sendn (So, m_buf, len2 );
If (AA <0 ){
Closesocket (so );
Break;
}
Len1 = len1-aa;
Len = len-AA;
　
}
Myfile. Close ();

}
The most important functions and technologies on the server end are described as follows:
The client is the most important and complex. It is responsible for thread management and progress recording.
The general process is as follows:
Connect to the server first, and then send command 1 (to give me the file information), including the file length, name, etc. Then, based on the length, it is decided to download the file in several threads and the initial download process, then, send command 2 (you can upload the file to me) and record the file process. Finally, finish.
A very important class is the cdownload class, which is defined as follows:
Class cdownload
{
Public:

　
Void createthread (); // open thread http://h2osky.126.com
DWORD finish1 (); // completion thread
Int sendlist (); // send command 1
Downinfo doinfo; // File Information (same as the server definition)
Int startask (int n); start to upload file n
Long m_index;
Bool good [Black];
Int filerange [100];
Cstring fname;
Cstring fnametwo;
Uint threadfunc (long index); // download process

Int sendrequest (int n); // File Sending information
Cdownload (INT thno1 );
Virtual ~ Cdownload ();

};

 

 

 

Next we will first introduce sendrequest (int n). Before starting,
Send a file message command to the server to let the client know which files can be uploaded.
Int cdownload: sendrequest (int n)
{
File: // create a socket
Sockaddr_in local;
Socket m_socket;

Int rc = 0;
File: // original server address
Local. sin_family = af_inet;
Local. sin_port = htons (1028 );
Local. sin_addr.s_un.s_addr = inet_addr (IP );
M_socket = socket (af_inet, sock_stream, 0 );

 
Int ret;
File: // connect to the server
Ret = connect (m_socket, (lpsockaddr) & Local, sizeof (local ));
File: // if there is an error
If (Ret <0 ){
Afxmessagebox ("connection error ");
Closesocket (m_socket );
Return-1;
}
File: // initial command
Fileinfo fileinfo1;
Fileinfo1.len = N;
Fileinfo1.seek = 50;
Fileinfo1.type = 1;
File: // send command
Int AA = sendn (m_socket, (char *) & fileinfo1, 100 );
If (AA <0) {closesocket (m_socket );
Return-1 ;}
File: // receives information from the server.
AA = readn (m_socket, (char *) & fileinfo1, 100 );
If (AA <0) {closesocket (m_socket );
Return-1 ;}
File: // close
Shutdown (m_socket, 2 );
Closesocket (m_socket );

Return 1;
}

With the file message, we can download the file.
In the main function, the usage is as follows:
File: // download the clno object and create a new cdownload class for it
Down [clno] = new cdownload (clno );
File: // start the download and make it available
Type = down [clno]-> startask (clno );
File: // create various threads
Createthread (clno );

Start Method
File: // start Method
Int cdownload: startask (int n)
{

File: // The length of the read object.
Doinfo. filelen = zmfile [N]. length;
　
File: // read the name
Fname = zmfile [N]. Name;
Cstring tmep;
File: // original file name
Tmep. Format ("// temp // % s", fname );

　

File: // send a message to the main function
Cstring AAA;
 
AAA = "reading" + fname + "information. Download now... /N ";
Afxgetmainwnd ()-> sendmessagetodescendants (wm_age1, (lparam) AAA. getbuffer (0), 1 );
AAA. releasebuffer ();
File: // if the file length is smaller than 0, return
If (doinfo. filelen <= 0) Return-1;
File: // create a file that ends with. Down to record file information
Cstring m_temp;
M_temp = fname + ". dwon ";
 
Doinfo. Name = m_temp;
File * fp = NULL;
Cfile myfile;
File: // if the file is downloaded for the first time

If (FP = fopen (m_temp, "R") = NULL ){
　

Filerange [0] = 0;
File: // file multipart
For (INT I = 0; I <black; I ++)
{
If (I> 0)
Filerange [I * 2] = I * (doinfo. filelen/Black + 1 );
　
Filerange [I * 2 + 1] = doinfo. filelen/Black + 1;
　
}
Filerange [Black * 2-1] = doinfo. filelen-filerange [Black * 2-2];

Myfile. Open (m_temp, cfile: modecreate | cfile: modewrite | cfile: typebinary );

　
File: // The length of the written file.
Myfile. Write (& doinfo. filelen, sizeof (INT ));
Myfile. Close ();
　
Cstring temp;
For (int ii = 0; II <black; II ++ ){
File: // indicates the information of each process record (ended with. downn)

Temp. Format (". Down % d", ii );
M_temp = fname + temp;
Myfile. Open (m_temp, cfile: modecreate | cfile: modewrite | cfile: typebinary );
File: // write the file information of each process
Myfile. Write (& filerange [II * 2], sizeof (INT ));
Myfile. Write (& filerange [II * 2 + 1], sizeof (INT ));
Myfile. Close ();
　
}

(Cmainframe *): afxgetmainwnd ()-> m_work.m_listctrl-> additemtwo (n, 2, 0, 0, doinfo. threadno );

 
}
File: // if the file already exists, it indicates that the upload is resumed and the last message is read.
Else
{
　
File: // open the file
Fread (& doinfo. filelen, sizeof (INT), 1, FP );

Fclose (FP );

Cstring temp;
　
M_temp = fname + ". down0 ";
If (FP = fopen (m_temp, "R") = NULL)
 
Return 1;
Else fclose (FP );
Int BB;
BB = 0;
File: // read the information recorded by each process
For (int ii = 0; II <black; II ++)
{
Temp. Format (". Down % d", ii );
M_temp = fname + temp;
　
Myfile. Open (m_temp, cfile: moderead | cfile: typebinary );
Myfile. Read (& filerange [II * 2], sizeof (INT ));
Myfile. Read (& filerange [II * 2 + 1], sizeof (INT ));
Myfile. Close ();
　

BB = BB + filerange [II * 2 + 1];
Cstring temp;
　

}
If (BB = 0) return 1;
Doinfo. totle = doinfo. filelen-BB;
　
　

　

　
　

(Cmainframe *): afxgetmainwnd ()-> m_work.m_listctrl-> additemtwo (n, 2, doinfo. totle, 1, 0, doinfo. threadno );

}

File: // create the download End Process timethread to check the end time of each process.
DWORD dwthread;
: Createthread (null, 0, timethread, (lpvoid) This, 0, & dwthread );

Return 0;
}
File: // The following describes the establishment of various process functions, very simple: http://h2osky.126.com
Void cmainframe: createthread (INT threadno)
{
DWORD dwthread;
File: // create a black Process
For (INT I = 0; I <black; I ++)
{
M_thread [threadno] [I] =: createthread (null, 0, downthread, (lpvoid) down [threadno], 0, & dwthread );
　

　
}

}
File: // downthread process function

DWORD winapi downthread (lpvoid lpparam)
{
Cdownload * pthis = (cdownload *) lpparam;
File: // process cabling + 1
Interlockedincrement (& pthis-> m_index );
File: // execute the download process
Pthis-> threadfunc (pthis-> m_index-1 );
Return 1;
}
The following describes how to download process functions.
Uint cdownload: threadfunc (long index)
{
File: // original join
Sockaddr_in local;
Socket m_socket;

Int rc = 0;
　
Local. sin_family = af_inet;
Local. sin_port = htons (1028 );
Local. sin_addr.s_un.s_addr = inet_addr (IP );
M_socket = socket (af_inet, sock_stream, 0 );

　
Int ret;
File: // read Cache
Char * m_buf = new char [size];
Int re, len2;
Fileinfo fileinfo1;
File: // join
Ret = connect (m_socket, (lpsockaddr) & Local, sizeof (local ));
File: // read the download information of each process
Fileinfo1.len = filerange [Index * 2 + 1];
Fileinfo1.seek = filerange [Index * 2];
Fileinfo1.type = 2;
Fileinfo1.fileno = doinfo. threadno;
　
Re = fileinfo1.len;
　

File: // open the file
Cfile destfile;
File * fp = NULL;
File: // if this is the first upload
If (FP = fopen (fname, "R") = NULL)

Destfile. Open (fname, cfile: modecreate | cfile: modewrite | cfile: typebinary | cfile: sharedenynone );

Else

File: // if the file exists, it is resumed.
Destfile. Open (fname, cfile: modewrite | cfile: typebinary | cfile: sharedenynone );
File: // move the file pointer to the specified position
Destfile. Seek (filerange [Index * 2], cfile: Begin );
File: // send the message to the server. You can upload the file.
Sendn (m_socket, (char *) & fileinfo1, 100 );

Cfile myfile;
File: // http://h2osky.126.com of the progress information of each process
Cstring temp;
Temp. Format (". Down % d", index );
M_temp = fname + temp;

File: // when the length of each segment is not 0
While (Re> 0 ){
　
　

Len2 = Re> size? Size: RE;
　
File: // read the content of each segment
Int len1 = readn (m_socket, m_buf, len2 );
File: // if there is an error
If (len1 <0 ){
Closesocket (m_socket );
Break;
}
　
File: // write the file
Destfile. Write (m_buf, len1 );

File: // modify the record progress

Filerange [Index * 2 + 1]-= len1;
Filerange [Index * 2] + = len1;
File: // move the record file pointer to the header
Myfile. Seek (0, cfile: Begin );
File: // write record progress
Myfile. Write (& filerange [Index * 2], sizeof (INT ));
Myfile. Write (& filerange [Index * 2 + 1], sizeof (INT ));

File: // minus the length of this read

Re = re-len1;

File: // Add the file length
Doinfo. totle = doinfo. totle + len1;
　

　
 
　

};
File: // the download is complete.
　
Myfile. Close ();
Destfile. Close ();
Delete [] m_buf;
Shutdown (m_socket, 2 );
　
　
If (Re <= 0) Good [Index] = true;
Return 1;
}
The main modules and Mechanisms of the client have been basically introduced. I hope to have a good understanding of this multi-threaded resumable data transfer method.
I have written a utility based on this knowledge (network transfer, including source code). It can be used on an IP/TCP-based computer for your learning. :
Http://h2osky.126.com
Zhao Ming
Email: papaya_zm@sina.com; zmpapaya@hotmail.com
Web: http://h2osky.126.com