Java.io.IOException:Filesystem closed

1. Description of the problem

When submitting a task to the cluster, you need to read a resource file on the HDFs. When reading the resource file, the code bursts the following exception:

Error:java.io.IOException:Filesystem closed at Org.apache.hadoop.hdfs.DFSClient.checkOpen (dfsclient.java:823) at Org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy (dfsinputstream.java:846) at Org.apache.hadoop.hdfs.DFSInputStream.read (dfsinputstream.java:907) at java.io.DataInputStream.readFully ( datainputstream.java:195) at java.io.DataInputStream.readFully (datainputstream.java:169) at Org.apache.parquet.had Oop. Parquetfilereader$consecutivechunklist.readall (parquetfilereader.java:756) at Org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup (parquetfilereader.java:494) at Org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead (internalparquetrecordreader.java:127) at Org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue (internalparquetrecordreader.java:208) at Org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue (parquetrecordreader.java:201) at Org.apache.hadoop.mapreduce.lib.input.DelegatingRecordReader. Nextkeyvalue (delegatingrecordreader.java:89) ... 

2. The cause of the problem

The API for general read HDFS files is this:

public void Initpackagenametypemap {
    Configuration conf = new Configuration ();
    FileSystem fs = Filesystem.get (conf);
    Path file = new Path ("FilePath");
    ...
}

When the task is submitted to the cluster, multiple datanode in the Getfilesystem process, as configuration, will get the same filesystem. If you have a datanode that closes the connection after you use it, the other datanode will receive the above exception. 3. The solution

The solution is actually very simple, add the following configuration in the code is good:

Conf.setboolean ("Fs.hdfs.impl.disable.cache", true);

There is a static cache inside the Filesytem class to hold the collection of instances of each file system, and the FileSystem class can specify whether to cache filesystem instances by "Fs.%s.impl.disable.cache" (where% S is replaced with the corresponding scheme, such as HDFs, Local, S3, S3n, and so on, once the corresponding filesystem instance is created, the instance is saved in the cache, and each get gets the same instance each time thereafter. So when set to true, the above exception can be resolved.

Reference Link:
1.http://stackoverflow.com/questions/23779186/ Ioexception-filesystem-closed-exception-when-running-oozie-workflow
2.http://stackoverflow.com/questions/ 20057881/hadoop-filesystem-closed-exception-when-doing-bufferedreader-close
3.http:// shift-alt-ctrl.iteye.com/blog/2108760