Java implementation of FTP batch large file upload download Chapter 1_java

This article describes how to use Java's existing available libraries to write FTP client code in Java, and develop applet controls to upload and download controls based on the web for bulk, large files. On the basis of comparing a series of FTP client libraries, this paper discusses some common functions such as progress bar, continuous transmission of breakpoint, mapping of internal and external network, callback JavaScript function in applet and so on, which is a more general and powerful J-ftp class library. Hope that through this article to play a role in the.

First, a primer

In the process of implementing a project, a web-based file upload and download requirement is presented. Users across the province (or across the country) need to upload some files to a file server in a center. These documents are used in a number of large-scale construction projects, may involve tens of millions of millions of construction projects. The file has three distinct characteristics: first, the file is large, may reach 50M, the second is the number of files, there are probably about 15; third, data security requires digital signature and data encryption.

The first consideration is the HTTP based transmission mode. But the author through relatively quickly found to meet the above requirements:

1: Upload with HTTP protocol, seems to be more suitable for web programming convenience, upload less than 1M file speed than with FTP protocol upload file slightly faster. But for the bulk and large file transmission may be powerless. Of course, it also has its advantages, if not like FTP, you must start an FTP service on the server side.

2: Upload files with FTP protocol larger than 1M files faster than HTTP. The larger the file, the faster the upload is than the HTTP upload speed several times. and write programs in Java, FTP is more convenient than HTTP.

I have used VB has also written ActiveX control to carry out batch file upload download, its function is also very powerful. Just because there is no special digital signature of the CAB file or OCX, it is necessary to set up a cumbersome client, such as setting up a secure site, reducing the security level of the client, and so on, thus giving up some schemes.

At the same time, taking into account the need for digital signature and data encryption on the client, decided to use the applet way to achieve. Before the file is uploaded, the client can obtain the local Usbkey key information and complete the encryption and signature processing of the uploaded file. Although the use of applet requirements for the client to install the JRE Runtime environment, the management and use of the client to bring a one-time inconvenience, but relative to such a large number of files and file security, this may be a relatively small price.

to sum up, the operating environment is:

FTP server side: Serv-u, professional FTP server-side program, the internet has off-the-shelf software downloads, of course, the reader may also write a server-side FTP file receiving program to explain. If there is no special requirements or features, serv-u should be able to meet the requirements of our general upload download;

Client: Java applet, that year let Java fire a so-called with the Microsoft ActiveX analogy Technology of course, now Java out of the JavaFX, is not a substitute for applets?

Application Environment: Internet network, the ultimate goal.

Second, the Java FTP client library selection

Let's imagine a situation where we want to write a pure Java application that uploads downloaded files from an FTP server running on a remote computer, and we also want to be able to get the basic file information, such as filename, data, or file size, for the remote files that are downloaded.

Although it is possible, and perhaps interesting, to write an FTP protocol handler from scratch, the task is difficult, lengthy, and potentially dangerous. Because we don't want to spend time, energy, or money to write a handler like this, we switch to a reusable component that already exists. And a lot of inventory is online.

Finding an excellent Java FTP client library that suits our needs is not as simple as it looks. On the contrary, it is a very painful and complicated task. First, it takes some time to find an FTP client library, and secondly, after we find all the existing libraries, which one should we choose? Each library is suitable for different needs. These libraries are not equivalent in performance, and they are fundamentally different in design. Each class library has its own features and uses different terms to describe them. Therefore, it is difficult to evaluate and compare the FTP client library.

The use of reusable components is a worthwhile approach, but in this case it is often daunting at first. Later, perhaps a little ashamed: After choosing a good FTP library, then the work is very simple, according to the simple rules to do. At present, there are a lot of open free FTP client class library, such as SIMPLEFTP, J-FTP, and many other ftpclient. As shown in the following table, the table is not fully listed, as readers have a better client FTP class library, please add further.

In this paper, the author adopts the j-ftp. This is an open source and powerful client FTP class library. I like it very much, but also recommend it to you readers. Forget to do a free advertisement for it.

Three, basic function

1, landing
using FTP for file transfer, in fact, the use of Java.net.socket to communicate. The following code is just one of the login methods for the class net.sf.jftp.net.FtpConnection. Of course in the following code, in order to save the layout, as well as some of the principles elaborated, the author will be some unnecessary code removed, such as the log code. Complete code refer to J-ftp's source code or the author's sample source, followed by the same code example:

	
  

		

   
public int login(String username, String password)
 {
  this.username = username;
  this.password = password;

  int status = LOGIN_OK;

  jcon = new JConnection(host, port);

  if(jcon.isThere())
  {
   in = jcon.getReader();

   if(getLine(POSITIVE) == null)//FTP220_SERVICE_READY) == null)
   {
    ok = false;    
    status = OFFLINE;
   } 

   if(!getLine(loginAck).startsWith(POSITIVE))//FTP230_LOGGED_IN))
   {    
    if(success(POSITIVE))//FTP230_LOGGED_IN))
    {     
    }
    else
    {
     ok = false;
     status = WRONG_LOGIN_DATA;
    }
   }
  }
  else
  {
   if(msg)
   {
    Log.debug("FTP not available!");
    ok = false;
    status = GENERIC_FAILED;
   }
  }

  if(ok)
  {
   connected = true;
   system();
   binary();
   
   String[] advSettings = new String[6];

   if(getOsType().indexOf("OS/2") >= 0)
   {
    LIST_DEFAULT = "LIST";
   }

   if(LIST.equals("default"))
   {
    //just get the first item (somehow it knows first is the
    //FTP list command)
    advSettings = LoadSet.loadSet(Settings.adv_settings);

    //*** IF FILE NOT FOUND, CREATE IT AND SET IT TO LIST_DEFAULT
    if(advSettings == null)
    {
     LIST = LIST_DEFAULT;

     SaveSet s = new SaveSet(Settings.adv_settings, LIST);
    }
    else
    {
     LIST = advSettings[0];

     if(LIST == null)
     {
      LIST = LIST_DEFAULT;
     }
    }
   }

   if(getOsType().indexOf("MVS") >= 0)
   {
    LIST = "LIST";
   }

   //***
   fireDirectoryUpdate(this);
   fireConnectionInitialized(this);
  }
  else
  {
   fireConnectionFailed(this, new Integer(status).toString());
  }

  return status;
 }


	
  
	

		

	

In this login method, there is a jconnection class that is responsible for establishing socket sockets    , and this class is a separate thread, the benefit of which is to accommodate changes in the interface, and the network socket connection and other work as a separate thread to deal with, conducive to the friendliness of the interface. The following is the run method for the Net.sf.jftp.net.JConnection class, and of course, the start of this thread is initiated in the construction method of the Jconnection class.

public void run()
 {
  try
  {
   s = new Socket(host, port);

   localPort = s.getLocalPort();

   //if(time > 0) s.setSoTimeout(time);
   out = new PrintStream(new BufferedOutputStream(s.getOutputStream(),
               Settings.bufferSize));
   in = new BufferedReader(new InputStreamReader(s.getInputStream()),
         Settings.bufferSize);
   isOk = true;

   // }
  }
  catch(Exception ex)
  {
   ex.printStackTrace();
   Log.out("WARNING: connection closed due to exception (" + host +
     ":" + port + ")");
   isOk = false;

   try
   {
    if((s != null) && !s.isClosed())
    {
     s.close();
    }

    if(out != null)
    {
     out.close();
    }

    if(in != null)
    {
     in.close();
    }
   }
   catch(Exception ex2)
   {
    ex2.printStackTrace();
    Log.out("WARNING: got more errors trying to close socket and streams");
   }
  }

  established = true;
 }

		
   

			
    

		
   

The socket in this run method illustrates that this class implements a client socket (or "socket"), which is the communication endpoint between two machines. The actual work of the socket is performed by an instance of the SocketImpl class. The application can configure itself to create sockets for the local firewall by changing the socket factory that creates the socket implementation. Please refer to JDK5 's API description for specific instructions, preferably in Chinese. Oh.

2 Upload Download
File Upload can be divided into multiple threads and single-threaded, in the case of single thread is simpler, and in the case of multithreading, to deal with things to be more points, but also to be careful a lot. Below is the Net.sf.jftp.net.FtpConnection upload Handleupload method. Two different types of single-threaded and multi-threaded have been considered.

public int handleUpload(String file, String realName)
 {
  if(Settings.getEnableMultiThreading() &&
    (!Settings.getNoUploadMultiThreading()))
  {
   Log.out("spawning new thread for this upload.");

   FtpTransfer t;

   if(realName != null)
   {
    t = new FtpTransfer(host, port, getLocalPath(), getCachedPWD(),
         file, username, password, Transfer.UPLOAD,
         handler, listeners, realName, crlf);
   }
   else
   {
    t = new FtpTransfer(host, port, getLocalPath(), getCachedPWD(),
         file, username, password, Transfer.UPLOAD,
         handler, listeners, crlf);
   }

   lastTransfer = t;

   return NEW_TRANSFER_SPAWNED;
  }
  else
  {
   if(Settings.getNoUploadMultiThreading())
   {
    Log.out("upload multithreading is disabled.");
   }
   else
   {
    Log.out("multithreading is completely disabled.");
   }

   return (realName == null) ? upload(file) : upload(file, realName);
  }
}

		
   

			
    

		
   

In the case of multithreading, there is a separate class net.sf.jftp.net. Ftptransfer, of course, in multi-threaded situations, this class must be a separate thread. Similar to Jconnection, the start of its thread is also initiated in the construction method. In its Run method, the file is read and transmitted.

 public void Run () {if (Handler.getconnections (). get (file) = null) {handler.addconnection (file, this);
   else if (!pause) {log.debug ("Transfer Already in progress:" + file);
   Work = false;

   Stat = 2;
  Return

  Boolean haspaused = false;

    while (pause) {try {runner.sleep (100); if (listeners!= null) {for (int i = 0; i < listeners.size (); i++) {((Connectionlistener) listeners
     . ElementAt (i)). updateprogress (file, paused,-1);
       } if (!work) {if (listeners!= null) {for (int i = 0; i < listeners.size (); i++) {
                       ((Connectionlistener) Listeners.elementat (i)). updateprogress (file, removed,
      -1);
  catch (Exception ex) {} haspaused = true; while ((Handler.getconnectionsize () >= settings.getmaxconnections ()) && (Handler.getconnecTionsize () > 0) && work) {try {stat = 4;

    Runner.sleep (400); if (!haspaused && (listeners!= null)) {for (int i = 0; i < listeners.size (); i++) {(Conne
     Ctionlistener) Listeners.elementat (i)). UpdateProgress (file, QUEUED,-1);
    } else {break;
   } catch (Exception ex) {ex.printstacktrace (); } if (!work) {if (listeners!= null) {for (int i = 0; i < listeners.size (); i++) {(Connect
    Ionlistener) Listeners.elementat (i)). updateprogress (file, removed,-1);
   } handler.removeconnection (file);

   Stat = 3;
  Return

  } started = true;
  try {runner.sleep (settings.ftptransferthreadpause);

  catch (Exception ex) {} con = new Ftpconnection (host, Port, RemotePath, CRLF);
  Con.setconnectionhandler (handler); Con.setconnectionlisteners (Listeners);

  int status = Con.login (user, pass);
   if (status = = FTPCONNECTION.LOGIN_OK) {File f = new file (LocalPath);

   Con.setlocalpath (F.getabsolutepath ());
    if (Type.equals (UPLOAD)) {if (newName!= null) {transferstatus = con.upload (file, newName);
    else {transferstatus = con.upload (file);
   } else {transferstatus = Con.download (file,this.newname);
  } if (!pause) {handler.removeconnection (file);

 }
 }
 public void run()
 {
  if(handler.getConnections().get(file) == null)
  {
   handler.addConnection(file, this);
  }
  else if(!pause)
  {
   Log.debug("Transfer already in progress: " + file);
   work = false;
   stat = 2;

   return;
  }

  boolean hasPaused = false;

  while(pause)
  {
   try
   {
    runner.sleep(100);

    if(listeners != null)
    {
     for(int i = 0; i < listeners.size(); i++)
     {
      ((ConnectionListener) listeners.elementAt(i)).updateProgress(file,
                      PAUSED,
                      -1);
     }
    }

    if(!work)
    {
     if(listeners != null)
     {
      for(int i = 0; i < listeners.size(); i++)
      {
       ((ConnectionListener) listeners.elementAt(i)).updateProgress(file,
                       REMOVED,
                       -1);
      }
     }
    }
   }
   catch(Exception ex)
   {
   }

   hasPaused = true;
  }

  while((handler.getConnectionSize() >= Settings.getMaxConnections()) &&
     (handler.getConnectionSize() > 0) && work)
  {
   try
   {
    stat = 4;
    runner.sleep(400);

    if(!hasPaused && (listeners != null))
    {
     for(int i = 0; i < listeners.size(); i++)
     {
      ((ConnectionListener) listeners.elementAt(i)).updateProgress(file,
                      QUEUED,
                      -1);
     }
    }
    else
    {
     break;
    }
   }
   catch(Exception ex)
   {
    ex.printStackTrace();
   }
  }

  if(!work)
  {
   if(listeners != null)
   {
    for(int i = 0; i < listeners.size(); i++)
    {
     ((ConnectionListener) listeners.elementAt(i)).updateProgress(file,
                     REMOVED,
                     -1);
    }
   }

   handler.removeConnection(file);
   stat = 3;

   return;
  }

  started = true;

  try
  {
   runner.sleep(Settings.ftpTransferThreadPause);
  }
  catch(Exception ex)
  {
  }

  con = new FtpConnection(host, port, remotePath, crlf);

  con.setConnectionHandler(handler);
  con.setConnectionListeners(listeners);

  int status = con.login(user, pass);

  if(status == FtpConnection.LOGIN_OK)
  {
   File f = new File(localPath);
   con.setLocalPath(f.getAbsolutePath());

   if(type.equals(UPLOAD))
   {
    if(newName != null)
    {
     transferStatus = con.upload(file, newName);
    }
    else
    {
     transferStatus = con.upload(file);
    }
   }
   else
   {
    transferStatus = con.download(file,this.newName);
   }
  }

  if(!pause)
  {
   handler.removeConnection(file);
  }
 }

As for the download process, because it is the reverse process upload, and upload methods and writing the same, in some space for the consideration, and did not list the code, but its ideas and ideas exactly the same. Please refer the source code to the reader.

Iv. progress bar

It can be imagined that if in the process of uploading or downloading, there is no hint, the user simply can not determine whether the task is completed or the task is dead, often because of the upload time or download time too long and misleading users. Therefore, the progress bar appears to be very important and practical.

The implementation of the progress bar, in fact, it is simple to say. is to open two threads in the program, the first thread is used to dynamically change the value of the progress bar on the interface, while the second thread in the upload or download process, make a loop, in this loop, each read a certain amount of data such as 8192 bytes. After this data is passed, the UpdateProgress method in the first thread is invoked to update the value of the interface progress bar.

In the process of uploading or downloading (see the Run method of the Ftptransfer class in the previous section), you can view the Con.upload (file, NewName) method, as shown in the following code,

 public int upload (string file, String realname, InputStream in) {hasuploaded = True
  ;

  Log.out ("FTP upload started:" + this);

  int stat;
   if (in = = null) && new file (file). Isdirectory ()) {shortprogress = true;
   FileCount = 0;
   Basefile = file;
   DataType = Dataconnection.putdir;

   Isdirupload = true;

   Stat = uploaddir (file);

   Shortprogress = false;
   System.out.println (FileCount + ":" + basefile);

   Fireprogressupdate (Basefile, dataconnection.dfinished + ":" + FileCount,-1);
   Fireactionfinished (this);
  Firedirectoryupdate (this);
   else {dataType = Dataconnection.put;

   Stat = rawupload (file, Realname, in);
   try {thread.sleep (100);
   The catch (Exception ex) {} fireactionfinished (this);
  Firedirectoryupdate (this);
  try {thread.sleep (500);
 The catch (Exception ex) {} return stat; }

This method is responsible for uploading a certain number of bytes of content, in fact, is called the Rawupload method, not listed here, please refer to the source code, and when the byte data passed through the call fireactionfinished () method to invoke the UpdateProgressBar () method in the main thread. In fact, the code is as follows:

 protected void UpdateProgressBar () {
  int percent = (int) ((float) lfilecompletesize/(float) lfilesize) * 10000F);pbfile.setvalue (percent);
  System.out.println ("=================================================" +percent);
  Pbfile.setstring (lfilecompletesize/1024l + "/" + lfilesize/1024l
    + "KB");
  Percent = (int) (((float) ltotalcompletesize/(float) ltotalsize) * 10000F);
  Pbtotal.setstring (ltotalcompletesize/1024l + "/" + ltotalsize/1024l
    + "KB");
  Pbtotal.setvalue (percent);
  Repaint ();
 }

 

With two progress bars, the first progress bar represents the current file upload or download progress, and the second progress bar represents the progress of all files being downloaded or uploaded. At the same time, in order to produce the movement of the progress bar or changes in the rate of progress is obvious, through Pbfile.setmaximum (10000) and Pbtotal.setmaximum (10000) The maximum of the progress bar set to 10000, rather than the usual we set 100. I think this is more good-looking, because sometimes when uploading or downloading because of network reasons, may change relatively small. If set to 100, the change is not particularly obvious.

Above is the ftp bulk file upload download basic article, I hope to help you learn, but also hope that we support the cloud-dwelling community.