Use Java to implement batch uploading and downloading of FTP large files

Abstract: This article describes how to use the existing available libraries of Java to compile FTP clients in Java.CodeAnd developed into an applet control, making it a web-based batch upload and download control for large files.ArticleCompare a series of detailed FTP libraries to help readers select the most appropriate one based on their own needs.
This article describes how to use the existing available libraries of Java to write FTP client code, develop it into an applet control, and create a batch upload and download Control Based on Web, large files. This article compares a series of detailed FTP libraries to help readers select the most appropriate one based on their own needs. At the same time, we will elaborate on some common functions, such as progress bars, resumable data transfer, ing between the Intranet and the Internet, and callback of JavaScript Functions in the applet, as well as code implementation, I hope that this article will serve as a reference.

I. Overview

During the implementation of a project, I encountered a Web-based file upload/download requirement. Users throughout the province (or throughout the country) need to upload some files to a central file server. These files are used in some large-scale engineering construction projects, which may involve tens of millions or even hundreds of millions of construction projects. The file has three distinctive features: 1. The file size may reach 50 MB; 2. the number of files is large, and about 15 files may exist; third, digital signatures and Data Encryption are required for data security.

The first consideration is the HTTP-based transmission mode. However, the author quickly found that the above requirements were met:

1: using HTTP to upload files seems to be more convenient for Web programming. Uploading files smaller than 1 MB is faster than uploading files using FTP. However, batch and large file transmission may be powerless. Of course, it also has its advantages. Unlike FTP, an FTP service must be started on the server.

2: uploading files larger than 1 MB over FTP is faster than HTTP. The larger the file, the upload speed is several times faster than the HTTP upload speed. And written in JavaProgramFTP is more convenient than HTTP.

I have used VB and ActiveX controls to upload and download files in batches, which is also very powerful. Because the cab file or OCX is not signed by a specific number, you need to perform complicated settings on the client, such as setting a secure site and reducing the security level of the client. Therefore, some solutions are abandoned.

At the same time, considering that the file needs to be digitally signed and encrypted on the client, it is decided to adopt the applet method .. Before uploading a file, you can obtain the local USB key information on the client to encrypt and sign the uploaded file. Although the use of Applet requires that the JRE runtime environment be installed on the client, it is inconvenient for the management and use of the client. However, the security of such a large number of files and files is relatively high, this may be a small cost.

The running environment is as follows:

FTP server: Serv-U, a Professional FTP server program, and ready-made software downloads on the Internet. Of course, readers may also write a server-side FTP file receiving program for explanation. If there are no special requirements or functions, Serv-U should be able to meet our general upload and download needs;

Client: Java Applet, a technology that once made Java a big fire and claimed to be comparable to Microsoft's ActiveX. Of course, Java now has javafx. Is it a substitute for applet?

Application Environment: Internet, the ultimate goal.

Ii. Selection of Java FTP client library

Let's assume that we want to write a Java-only application for uploading and downloading files from an FTP server running on a remote computer; we also hope to get the basic file information of the remote files for download, such as the file name, data, or file size.

Although it is possible to write an FTP protocol processor from the beginning and may be interesting, it is also difficult, long, and potentially dangerous. Because we don't want to spend time, energy, or money writing such a processing program in person, we turn to the reusable components that already exist. And a lot of inventory is on the Internet.

Finding an excellent Java FTP client library that suits our needs is not as simple as it looks. On the contrary, this is a very painful and complex task. First, it takes some time to find an FTP client database. Second, after we find all the existing databases, which one should we choose? Each database is suitable for different needs. These libraries are not equivalent in performance and have fundamental design differences. Each class library has its own characteristics and uses different terms to describe them. Therefore, it is difficult to evaluate and compare the FTP client database.

The use of reusable components is a recommendable method, but in this case, it is often discouraged at the beginning. Later, it may be a bit embarrassing: After selecting a good FTP database, the subsequent work will be very simple, just follow simple rules. Currently, there are many open and free FTP client class libraries, such as simpleftp and J-FTP, and many other ftpclient. As shown in the following table, not all of them are listed. If you have a better client ftp class library, please add it further.

FTP client class library name remarks
J-FTP http://jaist.dl.sourceforge.net/sourceforge/j-ftp/j-ftp-1.50.tar.gz.
Simpleftp http: // www.jibble.org/files/simpleftp.jar
Ftpclient com.w.isedt.net. FTP. ftpclient
Ftpprotocol com. IBM. Network. FTP. Protocol. ftpprotocol
Ftpconnection net.sf.jftp.net. ftpconnection
Ftpclient org.apache.commons.net. FTP. ftpclient
Ftpclient jshop. jnet. ftpclient
Ftpclient sun.net. FTP. ftpclient
FTP com. cqs. FTP. FTP
Ftp cz. DHL. FTP. FTP
Ftpclient org. Globus. Io. FTP. ftpclient

In this article, I use J-FTP. This is an open-source and powerful client ftp class library. I like it very much and recommend it to you. Make an advertisement for it for free.

III. Basic functions

1. Login

Using FTP for file transmission actually uses java.net. socket for communication. The following code is just a login method like net.sf.jftp.net. ftpconnection. Of course, in the following code, in order to save the layout and clarify some principles, the author removes unnecessary code, such as logs. For the complete code, see J-FTP'sSource codeOr the example source code of the author. The following code examples are also the same:

Public int login (string username, string password)
{
This. Username = username;
This. Password = password;
Int status = login_ OK;
Jcon = new jconnection (host, Port );
If (jcon. isthere ())
{
In = jcon. getreader ();
If (Getline (positive) = NULL) // ftp220_service_ready) = NULL)
{
OK = false;
Status = offline;
}
If (! Getline (loginack). startswith (positive) // ftp230_logged_in ))
{
If (success (positive) // ftp230_logged_in ))
{
}
Else
{
OK = false;
Status = wrong_login_data;
}
}
}
Else
{
If (MSG)
{
Log. debug ("FTP not available! ");
OK = false;
Status = generic_failed;
}
}
If (OK)
{
Connected = true;
System ();
Binary ();
String [] advsettings = new string [6];
If (getostype (). indexof ("OS/2")> = 0)
{
List_default = "list ";
}
If (list. Equals ("default "))
{
// Just get the first item (somehow it knows first is
// Ftp list Command)
Advsettings = loadset. loadset (settings. adv_settings );
// *** If file not found, create it and set it to list_default
If (advsettings = NULL)
{
List = list_default;
Saveset S = new saveset (settings. adv_settings, list );
}
Else
{
List = advsettings [0];
If (list = NULL)
{
List = list_default;
}
}
}
If (getostype (). indexof ("MVS")> = 0)
{
List = "list ";
}
//***
Firedirectoryupdate (this );
Fireconnectioninitialized (this );
}
Else
{
Fireconnectionfailed (this, new INTEGER (Status). tostring ());
}
Return status;
}

In this login method, there is a jconnection class, which is responsible for establishing socket sockets, and this class is a separate thread, the advantage is to cooperate with the interface changes, the network socket connection and other work are processed as separate threads, which is conducive to the friendliness of the interface. The following is the run method of the net.sf.jftp.net. jconnection class. Of course, the thread is started in the jconnection class constructor.

Public void run ()
{
Try
{
S = new socket (host, Port );
Localport = S. getlocalport ();
// If (Time> 0) S. setsotimeout (time );
Out = new printstream (New bufferedoutputstream (S. getoutputstream (),
Settings. buffersize ));
In = new bufferedreader (New inputstreamreader (S. getinputstream ()),
Settings. buffersize );
Isok = true;
//}
}
Catch (exception ex)
{
Ex. printstacktrace ();
Log. Out ("Warning: Connection closed due to exception (" + host +
":" + Port + ")");
Isok = false;
Try
{
If (S! = NULL )&&! S. isclosed ())
{
S. Close ();
}
If (OUT! = NULL)
{
Out. Close ();
}
If (in! = NULL)
{
In. Close ();
}
}
Catch (exception ex2)
{
Ex2.printstacktrace ();
Log. Out ("Warning: Got more errors trying to close socket and streams ");
}
}
Established = true;
} The socket in this run method describes the implementation of the client socket (also known as "socket"). The socket is the communication endpoint between two machines. The actual work of a socket is performed by an instance of the socketimpl class. By changing the socket factory implemented by creating a socket, an application can configure itself to create a socket suitable for the local firewall. For details, refer to the jdk5 API description, preferably in Chinese. Haha.

2. upload and download

File Upload can be divided into multiple threads and single threads, which is simple in single thread mode. In multi-thread mode, there are more things to handle, at the same time, be careful. The following describes the handleupload Method for uploading net.sf.jftp.net. ftpconnection. Two different types of single thread and multi-thread have been considered.

Public int handleupload (string file, string realname)
{< br> If (settings. getenablemultithreading () &
(! Settings. getnouploadmultithreading ()
{< br> log. out ("Spawning new thread for this upload. ");
ftptransfer t;
If (realname! = NULL)
{< br> T = new ftptransfer (host, port, getlocalpath (), getcachedpwd (),
file, username, password, transfer. upload,
handler, listeners, realname, CRLF);
}< br> else
{< br> T = new ftptransfer (host, port, getlocalpath (), getcachedpwd (),
file, username, password, transfer. upload,
handler, listeners, CRLF);
}< br> lasttransfer = T;
return new_transfer_spawned;
}< br> else
{
If (settings. getnouploadmultithreading ()
{< br> log. out ("Upload multithreading is disabled. ");
}< br> else
{< br> log. out ("multithreading is completely disabled. ");
}< br> return (realname = NULL )? Upload (File): Upload (file, realname);
}< BR >}

In the case of multithreading, there is a separate class net.sf.jftp.net. ftptransfer. Of course, in the case of multithreading, this class must be a separate thread. Similar to jconnection, the thread is started in the constructor. In its run method, the file is read and transmitted.

Public void run ()
{
If (handler. getconnections (). Get (File) = NULL)
{
Handler. addconnection (file, this );
}
Else if (! Pause)
{
Log. debug ("transfer already in progress:" + file );
Work = false;
Stat = 2;
Return;
}
Boolean haspaused = false;
While (pause)
{
Try
{
Runner. Sleep (100 );
If (listeners! = NULL)
{
For (INT I = 0; I <listeners. Size (); I ++)
{
(Connectionlistener) listeners. elementat (I). updateprogress (file,
Paused,
-1 );
}
}
If (! Work)
{
If (listeners! = NULL)
{
For (INT I = 0; I <listeners. Size (); I ++)
{
(Connectionlistener) listeners. elementat (I). updateprogress (file,
Removed,
-1 );
}
}
}
}
Catch (exception ex)
{
}
Haspaused = true;
}
While (handler. getconnectionsize ()> = settings. getmaxconnections ())&&
(Handler. getconnectionsize ()> 0) & Work)
{
Try
{
Stat = 4;
Runner. Sleep (400 );
If (! Haspaused & (listeners! = NULL ))
{
For (INT I = 0; I <listeners. Size (); I ++)
{
(Connectionlistener) listeners. elementat (I). updateprogress (file,
Queued,
-1 );
}
}
Else
{
Break;
}
}
Catch (exception ex)
{
Ex. printstacktrace ();
}
}
If (! Work)
{
If (listeners! = NULL)
{
For (INT I = 0; I <listeners. Size (); I ++)
{
(Connectionlistener) listeners. elementat (I). updateprogress (file,
Removed,
-1 );
}
}
Handler. removeconnection (File );
Stat = 3;
Return;
}
Started = true;
Try
{
Runner. Sleep (settings. ftptransferthreadpause );
}
Catch (exception ex)
{
}
Con = new ftpconnection (host, port, remotepath, CRLF );
Con. setconnectionhandler (handler );
Con. setconnectionlisteners (listeners );
Int status = con. login (user, pass );
If (status = ftpconnection. login_ OK)
{
File F = new file (localpath );
Con. setlocalpath (F. getabsolutepath ());
If (type. Equals (upload ))
{
If (newname! = NULL)
{
Transferstatus = con. Upload (file, newname );
}
Else
{
Transferstatus = con. Upload (File );
}
}
Else
{
Transferstatus = con. Download (file, this. newname );
}
}
If (! Pause)
{
Handler. removeconnection (File );
}
}

As for the download process, because it is the inverse process of the upload, it is similar to the upload method and writing method. For some space considerations, the Code is not listed, however, its thoughts and ideas are the same. Please refer to the source code.

4. progress bar

As you can imagine, if there are no prompts during the upload or download process, you cannot determine whether the task is completed or whether the task is dead, users are often misled due to the upload or download time being too long. Therefore, the progress bar is very important and practical.

The implementation of the progress bar is actually very simple. It is to enable two threads in the program. The first thread is used to dynamically change the value of the progress bar on the interface, while the second thread makes a loop during the upload or download process, in this cycle, a certain amount of data such as 8192 bytes is read each time. After the data is uploaded, call the updateprogress method in the first thread to update the value of the interface progress bar.

During the upload or download process (see the ftptransfer class run method in the previous section), you can view the con. Upload (file, newname) method. The Code is as follows,

Public int upload (string file, string realname, inputstream in)
{< br> hasuploaded = true;
log. out ("FTP upload started:" + this);
int Stat;
If (in = NULL) & New file (file ). isdirectory ()
{< br> progress SS = true;
filecount = 0;
basefile = file;
datatype = dataconnection. putdir;
isdirupload = true;
stat = uploaddir (File);
progress SS = false;
// system. out. println (filecount + ":" + basefile);
fireprogressupdate (basefile,
dataconnection. dfinished + ":" + filecount,-1);
fireactionfinished (this);
firedirectoryupdate (this );
}< br> else
{< br> datatype = dataconnection. put;
stat = rawupload (file, realname, In);
try
{< br> thread. sleep (100);
}< br> catch (exception ex)
{< BR >}< br> fireactionfinished (this );
firedirectoryupdate (this);
}< br> try
{< br> thread. sleep (500);
}< br> catch (exception ex)
{< BR >}< br> return Stat;
}

This method is used to upload a certain number of bytes of content. In fact, it is called the rawupload method, which is not listed here. Please refer to the source code. After transmitting the byte data, you can call fireactionfinished () to call the updateprogressbar () method in the main thread. The Code is as follows:

Protected void updateprogressbar (){
Int percent = (INT) (float) lfilecompletesize/(float) lfilesize) * 0000f );
Pbfile. setvalue (percent );
// System. out. println ("========================================== ============== "+ percent );
Pbfile. setstring (lfilecompletesize/1024l + "/" + lfilesize/1024l
+ "Kb ");
Percent = (INT) (float) ltotalcompletesize/(float) ltotalsize) * 0000f );
Pbtotal. setstring (ltotalcompletesize/1024l + "/" + ltotalsize/1024l
+ "Kb ");
Pbtotal. setvalue (percent );
Repaint ();
}

Two progress bars are used above. The first progress bars indicate the upload or download progress of the current file, and the second progress bars indicate the download or upload progress of all files. At the same time, in order to generate a progress bar to move or change the progress, the pbfile is used. setmaximum (10000) and pbtotal. setmaximum (10000) sets the maximum value of the progress bar to 10000, instead of the 100 we set at ordinary times. I think this is quite nice, because sometimes uploading or downloading may be slightly changed due to network reasons. If it is set to 100, the change is not particularly obvious.