Java implementation of FTP upload download: ftpclient class for FTP upload download large files (including leading to the phenomenon of suspended animation)

Introduction:

FtpClient is a powerful FTP upload download tool, you can achieve a variety of ways of FTP file transfer, you can support the upload download a variety of large files (has been used in practice), and the existence of the official website so that users convenient use of this tool and so on.

1. First, the program to set the FTP request mode is passive mode, that is, the program to request the FTP server, requesting the server to open a port, let the client transfer files. This is basic, but there is no way to avoid uploading blocking problems.

2. Second, set the connection timeout, data transmission timeout, etc., also can not avoid blocking.

3. Then, call the upload or download, call the Stream.Close () method, also can not avoid blocking, this is the basic operation, does not explain what.

4. Finally, I added the upload download listener (ftpclient own implementation of a listener), just to show the upload download progress.

Conclusion: 1. These are all basic functions, but if the program uploads or downloads a large file for a long time, or if many tasks are uploaded over a long period of time, there will be blocking.
Solution: The solution I'm using is to remove the listener and run it again, and the program stops blocking. The specific reason I am not very clear, may be related to the flow, make a note here,
I hope to help people who have the same problem with me.

2. If org.apache.commons.net.io.CopyStreamException:IOException caught while copying appears
or caused by:java.net.SocketException:Connection reset by Peer:socket write error exception,
You need to detect: (1) Whether the connection is closed in time, first logout, and then disconnection. (2) Whether the disk is full, this also occurs frequently.

Note: The attachment for your own FTP upload download tool, including their own write timeout listening function (with the resulting blocking problem)

</pre><pre name="code" class="java">import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.net.SocketException;
import java.util.HashMap;
import java.util.Map;
import org.apache.commons.net.ftp.FTPClient;
import org.apache.commons.net.ftp.FTPFile;
import org.apache.commons.net.ftp.FTPReply;
import org.apache.log4j.Logger;

import com.gg.service.vo.FtpTransferFileVo;

/**
 *
 * FTP related operation tools
 *
 * @author zzj
 * @date May 30, 2016 10:54:20 AM
 */
public class FtpUtil {

/**
* Log recorder
*/
private static Logger logger = Logger.getLogger(FtpUtil.class);

/**
* ftp server ip address
*/
private String ftpHost = "192.168.128.1";

/**
* ftp server login name
*/
private String ftpUserName = "pds";

/**
* ftp server login password
*/
private String ftpPassword = "<span style="font-family: Arial, Helvetica, sans-serif;">pds</span>";
To
/**
* The root directory where the ftp server is located
*/
private String ftpRootDir="";

/**
* ftp server port number
*/
private int ftpPort = 21;
To
/**
* The maximum number of days for file storage, 0 means permanent storage, other means to delete files after the specified storage days are reached
*/
private int maxStoreDays=0;
To
/**
* The size of the stream buffer
*/
private final static int UPLOAD_BUFFER_SIZE=1024*1024;

/**
* The total number of scans (total monitoring time=SCAN_NUMBER*ONE_SECOND*SCAN_SECOND)
*Currently: 5*60*(10S)=3000S=50Min
*/
public static final int SCAN_NUMBER = 5*60;

/**
* One second
*/
public static final int ONE_SECOND = 1000;

/**
* Each sleep time
*/
public static final int SCAN_SECOND = 10;


public static void main(String[] args) {
FtpUtil ftpUtil = new FtpUtil();
String absPath ="/home/test/aa2/aad/";
String fileString="D:/software/MyEclipse_10.7.zip";
//String url ="http://pc2-dx1.newasp.net/soft/good/eclipse-standard-kepler-SR1-win32.zip";
String urlString="http://192.168.92.7/cloud/v1/versionDFS/DCC/AL817_ROW_T_SECBOOT/hq6753_65u_b1o_l1/S1La40_USR_S275_1604251950_MP3V2_16G_ROW/S1La40_USR_S275_160425195
File tempFile = new File(urlString);
String fileName = tempFile.getName();
InputStream io =HttpClientUtil.getUrlInputStream(urlString);
try {
//io = new FileInputStream(new File(fileString));
ftpUtil.uploadHttpFile(io,absPath,fileName);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

/**
* Upload the network file of the specified remote URL to the ftp directory
* @param remoteStorDir ftp absolute path (directory + file name), this path must end with /
* @param url The absolute path of http where the file is located
* @author zzj
* @throws Exception
*/
public boolean uploadHttpFile(InputStream in,String remoteStorDir,String fileName) throws Exception {
To
logger.info("start uploading file to ftp...");
boolean result = false;
FTPClient ftpClient = null;
String absFilePath=null;
try {
To
// Create and log in to the ftp server
ftpClient = this.getFTPClient(ftpHost, ftpPassword, ftpUserName, ftpPort);
/*ftpClient.setDataTimeout(1000*1000);
ftpClient.setConnectTimeout(connectTimeout)*/
To
// Set up ftp upload progress listener
//ftpClient.setCopyStreamListener(createListener());
To
// Set PassiveMode passive mode-send a transmission request to the service
ftpClient.enterLocalPassiveMode();

// Set to transmit in binary stream
ftpClient.setFileType(FTPClient.BINARY_FILE_TYPE);
To
//Add timeout monitoring thread
new DemaonThread(ftpClient,fileName).start();
To
// handle directory operations
createDirs(ftpClient, remoteStorDir);
To
logger.info("being upload,please waiting...");
absFilePath = remoteStorDir+fileName;
To
// Finally, upload the io stream to the specified directory
result = ftpClient.storeFile(absFilePath, in);
in.close();
logger.info("the upload result is:"+result+" and file path:" +absFilePath);
} catch (Exception e) {
logger.error("upload file failed,", e);
//Delete temporary files that may be generated
if (absFilePath!=null) {
ftpClient.deleteFile(absFilePath);
}
throw new Exception(e);
} finally {
try {
/*if (!result && absFilePath!=null) {
ftpClient.deleteFile(absFilePath);
System.out.println("Delete successfully!");
}*/
ftpClient.logout();
if (ftpClient.isConnected()) {
ftpClient.disconnect();
ftpClient=null;
}
} catch (IOException e) {
logger.error("An exception occurred during the last FTP operation,",e);
}
}
return result;
}
To
/**
* Get FTPClient object
*
* @param ftpHost FTP host server
* @param ftpPassword FTP login password
* @param ftpUserName FTP login username
* @param ftpPort FTP port default is 21
* @return
*/
public FTPClient getFTPClient(String ftpHost, String ftpPassword, String ftpUserName, int ftpPort) {
FTPClient ftpClient = null;
try {
// Connect to FTP server
ftpClient = new FTPClient();
To
logger.info("start connect ftp server.");
ftpClient.connect(ftpHost);
To
//Log in to the ftp server
ftpClient.login(ftpUserName, ftpPassword);
ftpClient.setBufferSize(UPLOAD_BUFFER_SIZE);
To
			//overtime time
int defaultTimeoutSecond=30*60 * 1000;
ftpClient.setDefaultTimeout(defaultTimeoutSecond);
ftpClient.setConnectTimeout(defaultTimeoutSecond );
ftpClient.setDataTimeout(defaultTimeoutSecond);
To
logger.info("connect and login ftp server success.");
To
// Set the size of each upload
/*ftpClient.setBufferSize(UPLOAD_BUFFER_SIZE);*/
To
if (!FTPReply.isPositiveCompletion(ftpClient.getReplyCode())) {
logger.info("Not connected to FTP, username or wrong password. ");
ftpClient.logout();
ftpClient.disconnect();
ftpClient=null;
} else {
System.out.println("FTP connect success!");
}
} catch (SocketException e) {
logger.error("FTP IP address may be wrong, please configure it correctly.", e);
} catch (IOException e) {
logger.error("FTP port error, please configure it correctly.", e);
}
return ftpClient;
}

/**
* Monitor ftpclient timeout daemon thread
* @author zzj
* @date Jun 3, 2016 6:19:18 PM
*/
private class DemaonThread extends Thread {
private FTPClient ftpClient;
private String fileName;
int num = 0;
Long start = System.currentTimeMillis()/1000;
To
/**
* @param ftpClient2
* @param fileName
*/
public DemaonThread(FTPClient ftpClient2, String fileName) {
this.ftpClient = ftpClient2;
this.fileName=fileName;
}
@Override
public void run() {
while (num <SCAN_NUMBER && ftpClient.isConnected()) {
try {
System.out.println(fileName+",monitor ftpclient start..."+(System.currentTimeMillis()/1000-start)+" S." );
Thread.sleep(ONE_SECOND*SCAN_SECOND);
num++;
System.out.println(fileName+",monitor ftpclient timeout..." );
} catch (Exception e) {
e.printStackTrace();
}
}
try {
System.out.println(fileName+",monitor ftpclient timeout finish...");
ftpClient.logout();
if (ftpClient.isConnected()) {
ftpClient.disconnect();
ftpClient=null;
}
} catch (Exception e) {
System.out.println(fileName+",**********monitor happend error,"+e.getMessage());
}
}
}
To
/**
* Upload progress listener (may cause blocking)
* @return listener object
* @author zzj
*/
/*public CopyStreamListener createListener() {
return new CopyStreamListener() {
private long start = System.currentTimeMillis()/1000;
@Override
public void bytesTransferred(CopyStreamEvent event) {
System.out.println("transfeerred");
bytesTransferred(event.getTotalBytesTransferred(), event.getBytesTransferred(), event.getStreamSize());
}

@Override
public void bytesTransferred(long totalBytesTransferred, int bytesTransferred, long streamSize) {
System.out.println("Spended time: "+(System.currentTimeMillis()/1000-start)+" seconds.");
System.out.println("transfered total bytes:" + FileUtil.FormetFileSize(totalBytesTransferred) + ",per transfeerred types:" + bytesTransferred+",stream size="+streamSize);
}
};
}*/

/**
* Create the specified directory
* @param ftpClient ftp client
* @param remoteUpLoadPath ftp server directory
* @throws IOException
* @author zzj
*/
public static void createDirs(FTPClient ftpClient, String remoteUpLoadPath) throws IOException {
To
//Judge whether the directory exists layer by layer according to the path, if it does not exist, create it
//1. First enter the root directory of ftp
ftpClient.changeWorkingDirectory("/");
String[] dirs = remoteUpLoadPath.split("/");
for (String dir: dirs) {
//2. Create and enter a non-existent directory
if (!ftpClient.changeWorkingDirectory(dir)) {
int num = ftpClient.mkd(dir);
System.out.println(num);
ftpClient.changeWorkingDirectory(dir);
System.out.println("Enter the directory successfully:"+dir);
}
}
}

/**
* Download files from ftp
* @param sourceFtpFileDir The ftp directory where the file is located
* @param version The version number of the file to be obtained
* @return directory name after download
* @throws Exception
* @author zzj
*/
public FTPFile[] listFtpFiles(FTPClient ftpClient ,String sourceFtpFileDir) throws Exception{
logger.info("start downloading file from ftp...");
FTPFile[] ftpFiles = null;
try {
// Set up ftp upload progress listener
//ftpClient.setCopyStreamListener(createListener("downloading... "));
To
// Set PassiveMode passive mode-send a transmission request to the service
ftpClient.enterLocalPassiveMode();

// Set to transmit in binary stream
ftpClient.setFileType(FTPClient.BINARY_FILE_TYPE);
To
//Add timeout monitoring thread
new DemaonThread(ftpClient,"downloading... ").start();
To
//Get all the files in the specified directory, specify the suffix of the file to be obtained
ftpFiles =ftpClient.listFiles(sourceFtpFileDir, new FTPFileSuffixFilter(".rar,.zip"));
logger.info("list file size is:"+ftpFiles==null?0:ftpFiles.length);
} catch (Exception e) {
logger.error("download file failed,", e);
throw new Exception(e);
} finally {
}
return ftpFiles;
}
To
/**
* File stream of the file to be downloaded
* @param ftpClient ftp client object
* @param ftpFiles all files obtained
* @param version version also
* @return file stream
* @author zzj
* @throws IOException
*/
public Map<String, Object> downloadFtpFile(String fileDir,String version,FtpTransferFileVo transferFileVo) throws Exception{
Map<String, Object> map = new HashMap<String, Object>();
InputStream inputStream = null;
FTPClient ftpClient =null;
try {
ftpClient = this.getFTPClient(this.getFtpHost(), this.getFtpPassword(), this.getFtpUserName(),this.getFtpPort());
FTPFile[] ftpFiles=this.listFtpFiles(ftpClient, this.getFtpRootDir());
To
int num=0;
String fileName = null;
for(FTPFile file: ftpFiles){
String tn = file.getName();
tn = tn.substring(0,tn.lastIndexOf("."));
if (file.isFile()&& tn.equals(version)) {
fileName = file.getName();
num++;
}
}
if (num==0) {
throw new Exception("The file number of the corresponding version ["+version+"] was not found.");
}else if(num>1){
throw new Exception("The file corresponding to the version "+version+" is greater than 1, the number is: "+num+", please handle it manually");
}
To
//Set file path
transferFileVo.setFileName(fileName);
transferFileVo.setHttpFileUrl(fileDir+fileName);
To
boolean flag =ftpClient.changeWorkingDirectory(fileDir);
if (!flag) {
throw new Exception("The specified directory does not exist or ftp cannot be opened, the path is: "+fileDir);
}
System.out.println("Enter the directory: "+fileDir+" Result: "+flag );
To
//Execute download file
inputStream = ftpClient.retrieveFileStream(fileDir+fileName);
map.put("error","false");
} catch (Exception e) {
logger.error("An exception occurred,",e);
map.put("error", e.getMessage());
}finally{
map.put("stream", inputStream);
map.put("ftpClient", ftpClient);
}
return map;
    }
To
/**
* List of ftp directories corresponding to all factories
*/
public static Map<String,String> factoryMap = new HashMap<String, String>();
static{
factoryMap.put("ss", "ss");
}
To
/**
* Get the corresponding catalog according to the factory name
* @param factory factory name
* @return ftp subdirectory
* @author zzj
*/
public static String getFactoryDir(String factory){
String dirName=null;
for (Map.Entry<String,String> entry: factoryMap.entrySet()) {
String cname=entry.getKey();
if (factory.contains(cname)) {
dirName=entry.getValue();
break;
}
}
return dirName;
}
To
public String getFtpHost() {
return ftpHost;
}

public void setFtpHost(String ftpHost) {
this.ftpHost = ftpHost;
}

public String getFtpUserName() {
return ftpUserName;
}

public void setFtpUserName(String ftpUserName) {
this.ftpUserName = ftpUserName;
}

public String getFtpPassword() {
return ftpPassword;
}

public void setFtpPassword(String ftpPassword) {
this.ftpPassword = ftpPassword;
}

public int getFtpPort() {
return ftpPort;
}

public void setFtpPort(int ftpPort) {
this.ftpPort = ftpPort;
}

public String getFtpRootDir() {
return ftpRootDir;
}

public void setFtpRootDir(String ftpRootDir) {
this.ftpRootDir = ftpRootDir;
}

public int getMaxStoreDays() {
return maxStoreDays;
}

public void setMaxStoreDays(int maxStoreDays) {
this.maxStoreDays = maxStoreDays;
}
}

Package com.gg.service.util;

Import Org.apache.commons.net.ftp.FTPFile;
Import Org.apache.commons.net.ftp.FTPFileFilter;

/**
 * Filter file suffix filter
 * @author ZZJ
 * @date June 7, 2016 3:37:36 PM/public
class Ftpfilesuffixfilte R implements ftpfilefilter{

	/**
	 * sent over the suffix information (multiple times in English comma separated)
	 /
	private String filesuffix;
	
	/**
	 * initialization parameter
	 * @param subffix suffix *
	/Public Ftpfilesuffixfilter (String subffix) {
		This.filesuffix=subffix;
	}
	
	/* (Non-javadoc)
	 * @see org.apache.commons.net.ftp.ftpfilefilter#accept (org.apache.commons.net.ftp.FTPFile)
	 * *
	@Override public
	Boolean accept (Ftpfile file) {
		String filename = file.getname ();  
		String[] strings = Filesuffix.split (",");
		Boolean flag = false;
		for (String suf:strings) {
		  if (Filename.lastindexof (SUF)!=-1) {  
			 flag = true;
	      } 
		}
        Return flag
	}

}

commons-net can only achieve FTP, TFTP, FTPS and other types of FTP upload download, such as to achieve sftp upload download, please refer to Jsch This open source jar package, here is an example:

http://www.linux178.com/Java/ftp-sftp-progress.html?utm_source=tuicool&utm_medium=referral Progress Sample

Http://www.cnblogs.com/dongliyang/p/4173583.html Tool Class