Implement FTP resumable upload

 

Application requirements:

Network Disk development gradually enters the integration phase of each part. When a user modifies a file on the client or adds a new file, the file must be uploaded to the user directory corresponding to the server, therefore, three transmission modes are established for data transmission (namely, upload and download), namely FTP transmission, HTTP transmission, and UDT-based transmission. The three data transmission modes are configurable and can be called through different interfaces. Compared with these three methods, the UDT-based massive file transmission is worth researching and innovating. It is based on UDP at the underlying layer and achieves reliability control at the upper layer; at the same time, it fully considers the shortcomings of the congestion control algorithm for Transmission Based on TCP in the public network environment, and implements its own congestion control algorithm, in actual tests, the performance is also significantly higher than that of TCP-based transmission. We have only conducted technical research on the implementation of UDT file transmission and have not yet implemented it. This part of content will be mentioned in subsequent articles. In the past three days, only FTP-based files that support resumable upload and download are supported.

Implementation principle:

The application example of resumable data transfer closest to us is: thunder. When using thunder to download a large file, it implements the following functions: 1> after the computer suddenly loses power or the program suddenly exits, when we restart thunder, it will continue downloading the downloaded file from the point where the program exits, instead of downloading the file from the beginning. 2> you can set multiple threads to download files at the same time. Each thread only downloads a part of the file. For example, you can use three threads to download a 9000-byte file, the first thread downloads 1-3000 bytes, the second thread downloads 1-6000 bytes, and the third thread downloads 1-9000 bytes. The three threads download an object at the same time, but download different parts. It stores the downloaded file fragments at a certain location, when all three threads are downloaded, they are combined into a complete file. It is worth noting that its advantages are obvious.

In fact, the principle of resumable upload is very simple, that is, you can record the number of bytes that have been uploaded or downloaded in real time during upload or download. If the transmission is disconnected for some reason, when the download starts, you only need to re-download or upload from the downloaded location.

Resumable Data Transfer Using qftp:

In QT, there is an FTP implementation class: qftp, which provides the basic ftp usage, connecting to the FTP server: connecttohost; login: Login; upload: Put; download: get, using these methods, you can interact with the FTP server to upload and download files. However, the put and get methods provided by the native method cannot implement resumable data transfer. Therefore, to achieve resumable upload, we need to re-implement File Transfer and add the breakpoint resume control. In fact, the essence of FTP file transmission is to use TCP to achieve underlying file transmission. The general idea is: Use the connecttohost of qftp to log on to the FTP server, use login to log on to the FTP server, use rawcommand to send the original FTP command, and use qtcpsocket to transmit file data.

First, to use qtcpsocket for file data transmission, you must set the FTP server to "PASV" for passive receiving, that is, the FTP server passively receives connection requests from clients.

All native commands that can be sent by the FTP server include: http://www.nsftools.com/tips/rawftp.htm.

The command sending process for breakpoint upload is as follows:

1. rawcommand ("type I"); set the data transmission type: binary data or ASCII

2. rawcommand ("PASV"); set the server to a passive receiving mode. After the PASV command is sent, the server returns its own data transmission port, waiting for the client to connect for data transmission. The returned data is in the format of "227 entering passive mode (192,168, 2, 18,118, 32)", and then the related information is included in the returned information. IP address of the FTP server: 192.168.2.18; port opened by the FTP server for data transmission: 118*256 + 32 = 30240. After obtaining this information, you need to establish a tcpsocket communication link to connect to the FTP server.

3. rawcommand ("appe remote-file-path"); Set remote-file-path on the server to append. If the file does not exist, the server creates one.

4. After completing the above process, you can open the local file for reading and send it through the tcpsocket Link (write ).

The command sending process for breakpoint download is as follows:

1. rawcommand ("type I"); set the data transmission type: binary data or ASCII

2. rawcommand ("PASV"); set the server to a passive receiving mode. After the PASV command is sent, the server returns its own data transmission port, waiting for the client to connect for data transmission. The returned data is in the format of "227 entering passive mode (192,168, 2, 18,118, 32)", and then the related information is included in the returned information. IP address of the FTP server: 192.168.2.18; port opened by the FTP server for data transmission: 118*256 + 32 = 30240. After obtaining this information, you need to establish a tcpsocket communication link to connect to the FTP server.

3. rawcommand ("Rest size"); this command sets the place where the FTP server starts data transmission from the local file.

4. rawcommand ("RETR remote-file-path"); transfers files from the remote host.

When a file is uploaded, you can open the local file for upload after setting the appe response. When downloading the file, you need to establish a connection to the tcpsocket after receiving the PASV response information. readyread () to read data.

Key code:

1. process the slot function after rawcommand () sends the native command return:

 

[CPP]View plaincopy

1. Void lhtfiletransfer: procrawcommandreply (INT nreplycode, qstring strdetail)
2. {
3. //! Type
4. If (200 = nreplycode)
5. {
6. M_ftphandle-> rawcommand ("PASV ");
8. If (currentitem. task_type.compare ("Upload") = 0)
9. {
10. OP = qstring ("put ");
11. }
12. Else if (currentitem. task_type.compare ("Download") = 0)
13. {
14. OP = qstring ("get ");
15. }
16. }
18. //! PASV
19. Else if (227 = nreplycode)
20. {
21. Const qstring backresult = strdetail;
23. If (null! = M_senddatasocket)
24. {
25. M_senddatasocket-> close ();
26. Delete m_senddatasocket;
27. }
29. M_senddatasocket = new qtcpsocket ();
31. Connect (m_senddatasocket, signal (readyread (), this, slot (procreadyread (), QT: uniqueconnection );
32. Connect (m_senddatasocket, signal (readchannelfinished (), this, slot (procreadchannelfinished (), QT: uniqueconnection );
33. Connect (m_senddatasocket, signal (byteswritten (qint64), this, slot (procbyteswritten (qint64), QT: uniqueconnection );
35. Qstringlist lstr = backresult. Split ("("). Last (). Split (")"). First (). Split (",");
36. Int naddress = lstr. At (0). toint () <24 |
37. Lstr. at (1). toint () <16 |
38. Lstr. At (2). toint () <8 |
39. Lstr. At (3). toint ();
40. Qhostaddress hostaddress (naddress );
41. Int nport = lstr. At (lstr. Length ()-2). toint () * 256 + lstr. Last (). toint ();
42. M_senddatasocket-> connecttohost (hostaddress, nport );
44. //! APPE, the absolute path of the remote file to be connected
45. Qstring appeshell;
46. If (op. Compare ("put") = 0)
47. {
48. Appeshell = qstring ("appe % 1"). Arg (currentitem. file_remote_path );
49. }
50. Else if (op. Compare ("get") = 0)
51. {
52. //! The size after rest here should be the size of the local saved price.
53. Appeshell = qstring ("Rest 0 ");
54. }
56. M_ftphandle-> rawcommand (appeshell );
57. }
59. //! Send data
60. Else if (150 = nreplycode)
61. {
62. If (op. Compare ("put") = 0)
63. {
64. M_filehandle = new qfile (currentitemfilepath );
66. If (! M_filehandle-> open (qiodevice: readonly ))
67. {
68. Qdebug () <"file open error ...";
69. Return;
70. }
72. Const qint64 filesize = m_filehandle-> size ();
74. M_filehandle-> seek (currentitem. uploaded_size );
76. While (! M_filehandle-> atend ())
77. {
78. Const qint64 nblocksize = 16*1024;
79. Char Buf [16*1024];
81. Qint64 nowpos = m_filehandle-> pos ();
83. Qint64 readlen = m_filehandle-> Read (BUF, nblocksize );
85. If (readlen! = 0 & readlen! =-1)
86. {
87. M_senddatasocket-> write (BUF, readlen );
88. M_senddatasocket-> flush ();
89. Emit datatransferprogress (nowpos, filesize );
90. }
91. }
93. M_senddatasocket-> flush ();
94. M_senddatasocket-> close ();
95. M_senddatasocket = NULL;
97. Emit datatransferprogress (m_filehandle-> pos (), m_filehandle-> size ());
99. M_proctask.remove (currentitemfilepath );
101. M_filehandle-> close ();
103. // Emit startnexttask ();
105. }
106. Else if (op. Compare ("get") = 0)
107. {
108. M_filehandle = new qfile (currentitem. file_remote_path );
110. If (! M_filehandle-> open (qiodevice: writeonly ))
111. {
112. Qdebug () <"file open error ...";
113. Return;
114. }
115. }
116. }
118. Else if (350 = nreplycode)
119. {
120. Qstring shell = qstring ("retr % 1"). Arg (currentitemfilepath );
121. M_ftphandle-> rawcommand (Shell );
122. }
123. }

2. Implement the buffer READ function during breakpoint download:

 

 

[CPP]View plaincopy

1. Void lhtfiletransfer: procreadyread ()
2. {
3. Qdebug () <"[Download] procreadyready ....";
4. Qbytearray buffer = m_senddatasocket-> readall ();
5. M_filehandle-> write (buffer );
7. M_filehandle-> flush ();
9. Emit datatransferprogress (m_filehandle-> size (), 0 );
10. }

Problems and problems to be optimized in the future:

 

1. character encoding problem, that is, when the file name to be uploaded is a Chinese name, it needs to be transcoded.

2. The current implementation is a single thread, and multi-thread breakpoint download and queue implementation have not yet been added.

Implement FTP resumable upload