Apache Ftpclient.listfiles () return empty problem discussion resolution

Occasionally I also met the Apache Ftpclient.listfiles () to get the file empty problem.

Target server environment: HP minicomputer

Client Server environment: Linux jstmsapp2 2.6.32-279.el6.x86_64 #1 SMP Wed June 18:24:36 EDT x86_64 x86_64 x86_64 (scripts in this dress Execution on the device)

Related Jar:common-net-1.4.1.jar (Common-net-3.3.jar still have this problem), Jakarta-oro-2.0.8.jar

My code is as follows:

   /**
     * @desc: FTP to get files from the target server to the local
     * @author<chengsheng.wang@zznode.com>
     * @since 2015-7-27
     *
     * @param url
     * @param userName
     * @param password
     * @param portnum
     * @param path
     * @param localPath
     * @return boolean
     */
    private boolean downLoadFromFtp(String url, String userName, String password,int portnum ,String path, String localPath){
    logger.info("url=" + url + "username=" + userName +" password=" + password + "hostpath=" + path +" localpath=" + localPath);
    boolean flag = false;
    FTPClient ftpClient = new FTPClient();
ftpClient.setControlEncoding("GBK");
int count = 0;//Sync file count
try {
ftpClient.connect(url, portnum);
boolean loginFlag = ftpClient.login(userName, password);
logger.info("Login status:"+loginFlag);
ftpClient.changeWorkingDirectory(path);
ftpClient.enterLocalPassiveMode();
FTPFile[] files = ftpClient.listFiles();
if (null == files || files.length == 0) {
logger.info("No file data");
return flag;
}
File tempfile = null;
FileOutputStream fos = null;
File localpathdir = new File(localPath);
if (!localpathdir.exists()) {
localpathdir.mkdirs();
}
logger.info("Total number of host directory files:"+files.length);
for (int i = 0; i <files.length; i++) {
if(files[i] == null){
continue;
}
String fileName = files[i].getName();
logger.info("th" + i + "file name:" + fileName);
String local = localPath + File.separator + fileName;
tempfile = new File(local);
if(tempfile.exists()){
continue;//If the file already exists, download/synchronize will not be repeated
}
fos = new FileOutputStream(tempfile);
ftpClient.setBufferSize(1024);
ftpClient.setFileType(FTPClient.BINARY_FILE_TYPE);
ftpClient.retrieveFile(path + File.separator + fileName, fos);
fos.close();
count++;
}
flag = true;
} catch (SocketException e) {
logger.error("Socket exception", e);
} catch (IOException e) {
logger.error("IO exception", e);
} catch (Exception e) {
logger.error("ftp download file abnormal", e);
}finally{
if (null != ftpClient) {
try {
if (ftpClient.isConnected()) {
ftpClient.logout();
ftpClient.disconnect();
}
} catch (IOException e) {
logger.error("Close connection exception", e);
}
}
logger.info("A total of "+count+" files have been synchronized this time");
}
    return flag;
    }

Execution to Ftpclient.listfiles (), Dead or dead returns empty.

Online research for a long time, inspired by some predecessors, speculated that the reason is the target server's Chinese language environment, resulting in file modification date format, can not be the correct analysis of Apache caused.

From the Internet to find Common-net-1.4.1.jar source code: Http://apache.fayea.com//commons/net/source/commons-net-1.4.1-src.zip

In the source code directly into the log debugging, and then Ftpclient.listfiles () return null problem suddenly enlightened.

Common-net-1.4.1.jar in the question, come one by one to explain:

Unixftpentryparser.java in Parseftpentry

   /**
     * Parses a line of a unix (standard) FTP server file listing and converts
     * it into a usable format in the form of an <code> FTPFile </code>
     * instance. If the file listing line doesn't describe a file,
     * <code> null </code> is returned, otherwise a <code> FTPFile </code>
     * instance representing the files in the directory is returned.
     * <p>
     * @param entry A line of text from the file listing
     * @return An FTPFile instance corresponding to the supplied entry
     */
public FTPFile parseFTPEntry(String entry) {
        FTPFile file = new FTPFile();
        file.setRawListing(entry);
        int type;
        boolean isDevice = false;

        if (matches(entry))//There is also a problem with the regular expression matching file information here. It is written to death. Its matching rules cause some files to be filtered because of the last modified date information.
        {
            String typeStr = group(1);
            String hardLinkCount = group(15);
            String usr = group(16);
            String grp = group(17);
            String filesize = group(18);
            String datestr = group(19) + "" + group(20);
            String name = group(21);
            String endtoken = group(22);

            try
            {
                file.setTimestamp(super.parseTimestamp(datestr)); //The problem lies here. The file date format caused by the locale cannot be parsed, which leads to return null and hides the abnormal information of the parsing error
            }
            catch (ParseException e)
            {
            return null; // this is a parsing failure too.
            } 

The regular expression with the problem

/**
     * this is the regular expression used by this parser.
     *
     * Permissions:
     * r the file is readable
     * w the file is writable
     * x the file is executable
     * - the indicated permission is not granted
     * L mandatory locking occurs during access (the set-group-ID bit is
     * on and the group execution bit is off)
     * s the set-user-ID or set-group-ID bit is on, and the corresponding
     * user or group execution bit is also on
     * S undefined bit-state (the set-user-ID bit is on and the user
     * execution bit is off)
     * t the 1000 (octal) bit, or sticky bit, is on [see chmod(1)], and
     * execution is on
     * T the 1000 bit is turned on, and execution is off (undefined bit-
     * state)
     */
    Private static final String REGEX =
        "([bcdlfmpSs-])"
        +"(((r|-)(w|-)([xsStTL-]))((r|-)(w|-)([xsStTL-]))((r|-)(w|-) ([xsStTL-])))\\+?\\s+"
        + "(\\d+)\\s+"
        + "(\\S+)\\s+"
        + "(?:(\\S+)\\s+)?"
        + "(\\d+)\\s+"
        
        /*
          Numeric or standard format date
        */
        + "((::\\d+[-/]\\d+[-/]\\d+)|(?:\\S+\\s+\\S+))\\s+" //This is a problem, Some files have been filtered, but the modified date of some files on the hp machine is also incredible.

        /*
           Year (for non-recent standard format)
Or time (for numeric or recent standard format
*/
+ "(\\d+(?::\\d+)?)\\s+"
        
+ "(\\S*)(\\s*.*)";

Now that you know the cause of the problem, discuss the solution

The internet has predecessors succinctly pointed out that the two places have been modified not on the line.

If you don't care about the last modification of the file, the most convenient thing to do is:

File.settimestamp (Calendar.getinstance ()); Resetting the last modification of the file to the current time is nothing wrong.

The regular expression can also be divert to read: "(?: \ \d+[-/]\\d+[-/]\\d+) | (?:\ \s+\\s+\\s+) | (?:\ \s+)) \\s+ "

Recompile a new common-net.1 ... 4.1.jar and then execute again, the world finally peace, all good.

Please refer to: http://www.blogjava.net/wodong/archive/2008/08/21/wodong.html

But is this really good? Elegant. The Apache contributors ' code still has room for us to refine the bug.

Step through the code from the Org.apache.commons.net.ftp.FTPClient.listFiles () method.

Listfiles () finally called the

Public ftpfile[] Listfiles (String pathname)
    throws IOException
    {
        string key = null;
        Ftplistparseengine engine =
            initiatelistparsing (key, pathname);
        return Engine.getfiles ();

    }

Then we continue to analyze initiatelistparsing (key, pathname)

  public FTPListParseEngine initiateListParsing(
            String parserKey, String pathname)
    throws IOException
    {
        // We cache the value to avoid creation of a new object every
        // time a file listing is generated.
        if(__entryParser == null) {
            if (null != parserKey) {
                // if a parser key was supplied in the parameters, 
                // use that to create the paraser
        	    __entryParser = 
        	        __parserFactory.createFileEntryParser(parserKey);
                
            } else {
	            // if no parserKey was supplied, check for a configuration
	        	// in the params, and if non-null, use that.
            	if (null != __configuration) {
            	    __entryParser = 
            	        __parserFactory.createFileEntryParser(__configuration);
            	} else {
                    // if a parserKey hasn't been supplied, and a configuration
            	    // hasn't been supplied, then autodetect by calling
                    // the SYST command and use that to choose the parser.
            	    __entryParser = 
            	        __parserFactory.createFileEntryParser(getSystemName());
             	}
            }
        }

        return initiateListParsing(__entryParser, pathname);

    }

It turns out that the __entryparser can be initialized by __configuration parameters. The default __configuration is NULL, which causes the program to execute to the

__entryparser = __parserfactory.createfileentryparser (Getsystemname ()); Initializes a parser that does not support body format

Continue to assume that we have a new ftpclientconfig, initialize parser by Ftpclientconfig, and continue with the code

	Public Ftpfileentryparser createfileentryparser (ftpclientconfig config) 
	throws Parserinitializationexception 
	{
	    this.config = config;
		String key = Config.getserversystemkey ();
		return Createfileentryparser (key);
	}

Enter the Createfileentryparser (key) method to reveal the ultimate truth

public FTPFileEntryParser createFileEntryParser (String key)
    {
        Class parserClass = null;
        FTPFileEntryParser parser = null;
        try
        {
            parserClass = Class.forName (key); // If we use the key to initialize a custom FTPFileEntryParser, is it OK, the key is passed from FTPClientConfig
            parser = (FTPFileEntryParser) parserClass.newInstance ();
        }
        catch (ClassNotFoundException e)
        {
            String ukey = null;
            if (null! = key)
            {
                ukey = key.toUpperCase ();
            }
            if (ukey.indexOf (FTPClientConfig.SYST_UNIX)> = 0)
            {
                parser = createUnixFTPEntryParser ();
            }
            else if (ukey.indexOf (FTPClientConfig.SYST_VMS)> = 0)
            {
                parser = createVMSVersioningFTPEntryParser ();
            }
            else if (ukey.indexOf (FTPClientConfig.SYST_NT)> = 0)
            {
                parser = createNTFTPEntryParser ();
            }
            else if (ukey.indexOf (FTPClientConfig.SYST_OS2)> = 0)
            {
                parser = createOS2FTPEntryParser ();
            }
            else if (ukey.indexOf (FTPClientConfig.SYST_OS400)> = 0)
            {
                parser = createOS400FTPEntryParser ();
            }
            else if (ukey.indexOf (FTPClientConfig.SYST_MVS)> = 0)
            {
                parser = createMVSEntryParser ();
        }
            else
            {
                throw new ParserInitializationException ("Unknown parser type:" + key);
            }
        }
        catch (ClassCastException e)
        {
            throw new ParserInitializationException (parserClass.getName ()
                + "does not implement the interface"
                + "org.apache.commons.net.ftp.FTPFileEntryParser.", e);
        }
        catch (Throwable e)
        {
            throw new ParserInitializationException ("Error initializing parser", e);
        }

        if (parser instanceof Configurable) {
            ((Configurable) parser) .configure (this.config);
        }
        return parser;
    } 

Careful netizen must have found that we can set a ftpclientconfig by giving the FtpClient object,

Through the Ftpclientconfig Systemkey attribute, initialize a custom ftpfileentryparser to complete the parsing work of the file time and other information.

Let's see if Ftpclientconfig has a constructor that meets the requirements.

	/**
	 * The main constructor for a Ftpclientconfig object
	 * @param systemkey key representing system type of THEserver being 
	 * connected to. @link #getServerSystemKey () serversystemkey} */
	 public
	ftpclientconfig (String systemkey) {
		This.serversystemkey = Systemkey;
	}

Well, there's just one constructor that can be started.

Create a new Unixftpentryparser, inherit from Configurableftpfileentryparserimpl, and then in Ftpclient.listfiles (); Before the call, Initializes a ftpclientconfig to the FtpClient object.

Look at the code:

Ftpclient.changeworkingdirectory (path);
			Ftpclient.enterlocalpassivemode (); 
			Because Apache does not support the Chinese language environment, a custom class resolves the Chinese date type
			ftpclient.configure (New Ftpclientconfig (" Com.zznode.tnms.ra.c11n.nj.resource.ftp.UnixFTPEntryParser "));
			ftpfile[] files = ftpclient.listfiles ();

Finally ended, very tired, the end of the attached custom Unixftpentryparser.java and Ftptimestampparserimplexzh.java (used to deal with the Chinese date, do not care about the date of modification can also not use it):

http://download.csdn.net/detail/wangchsh2008/8939331

This article to solve the problem of reference online predecessors, the solution by my actual application verification available, special post for netizens reference.