About the java.nio.channels.ClosedByInterruptException of Hadoop

About Java.io.IOException:Call to master/10.200.187.77:8020 failed on the local exception: Java.nio.channels.ClosedByInterruptException abnormal problem,

Recently changed the Hadoop log to debug mode for careful analysis, the following log error information interception section:

Note the Red Font section: This information closes the 77:40,956 connection, causing the back write to be thrown unexpectedly.

2012-06-28 14:12:20,433 DEBUG Org.apache.hadoop.ipc.Server:IPC Server handler 289 on 8020:has #4952 from 10.200.187.77:4 0956
2012-06-28 14:12:20,434 INFO org.apache.hadoop.hdfs.statechange:block* namesystem.addtoinvalidates:blk_- 6167110925703343645 is added to Invalidset of 10.200.187.78:50010
2012-06-28 14:12:20,434 INFO org.apache.hadoop.hdfs.statechange:block* namesystem.addtoinvalidates:blk_- 6167110925703343645 is added to Invalidset of 10.200.187.80:50010
2012-06-28 14:12:20,434 INFO org.apache.hadoop.hdfs.statechange:block* namesystem.addtoinvalidates:blk_- 6167110925703343645 is added to Invalidset of 10.200.187.79:50010
2012-06-28 14:12:20,435 DEBUG Org.apache.hadoop.ipc.Server:got #4905
2012-06-28 14:12:20,435 DEBUG org.apache.hadoop.ipc.Server:IPC Server Listener on 8020:disconnecting client 10.200. 187.77:40956. Number of active connections:105
2012-06-28 14:12:20,435 DEBUG Org.apache.hadoop.ipc.Server:IPC Server handler 321 on 8020:has #4905 from 10.200.187.77:4 0956
2012-06-28 14:12:20,435 DEBUG org.apache.hadoop.ipc.Server:Served:getProtocolVersion queuetime= 0 procesingtime= 0
2012-06-28 14:12:20,435 DEBUG Org.apache.hadoop.ipc.Server:IPC Server responder:responding to #4905 from 10.200.187.77: 40956
2012-06-28 14:12:20,435 WARN Org.apache.hadoop.ipc.Server:IPC Server Responder, call Getprotocolversion ( Org.apache.hadoop.hdfs.protocol.ClientProtocol from 10.200.187.77:40956:output Error
2012-06-28 14:12:20,435 INFO Org.apache.hadoop.ipc.Server:IPC Server handler 321 on 8020 Caught:java.nio.channels.Close Dchannelexception
At Sun.nio.ch.SocketChannelImpl.ensureWriteOpen (socketchannelimpl.java:133)
At Sun.nio.ch.SocketChannelImpl.write (socketchannelimpl.java:325)
At Org.apache.hadoop.ipc.Server.channelWrite (server.java:1700)
At org.apache.hadoop.ipc.server.access$2000 (server.java:94)
At Org.apache.hadoop.ipc.server$responder.processresponse (server.java:741)
At Org.apache.hadoop.ipc.server$responder.dorespond (server.java:804)
At Org.apache.hadoop.ipc.server$handler.run (server.java:1416)

2012-06-28 14:12:20,445 DEBUG org.apache.hadoop.ipc.Server:Served:setReplication queuetime= 0 procesingtime= 13
2012-06-28 14:12:20,445 DEBUG Org.apache.hadoop.ipc.Server:IPC Server responder:responding to #4950 from 10.200.187.77: 40956
2012-06-28 14:12:20,445 WARN Org.apache.hadoop.ipc.Server:IPC Server Responder, call Setreplication (/tmp/hive-hadoop/ HIVE_2012-06-28_14-09-10_170_1995016130043493038/-MR-10069/6E309161-EB89-4DBE-A6CB-EA3CB0167EE0) from 10.200.187.77:40956:output Error
2012-06-28 14:12:20,445 INFO Org.apache.hadoop.ipc.Server:IPC Server handler 328 on 8020 caught:java.nio.channels.Close Dchannelexception
At Sun.nio.ch.SocketChannelImpl.ensureWriteOpen (socketchannelimpl.java:133)
At Sun.nio.ch.SocketChannelImpl.write (socketchannelimpl.java:325)
At Org.apache.hadoop.ipc.Server.channelWrite (server.java:1700)
At org.apache.hadoop.ipc.server.access$2000 (server.java:94)
At Org.apache.hadoop.ipc.server$responder.processresponse (server.java:741)
At Org.apache.hadoop.ipc.server$responder.dorespond (server.java:804)
At Org.apache.hadoop.ipc.server$handler.run (server.java:1416)

2012-06-28 14:12:20,477 DEBUG org.apache.hadoop.ipc.Server:Served:delete queuetime= 0 procesingtime= 44
2012-06-28 14:12:20,477 DEBUG Org.apache.hadoop.ipc.Server:IPC Server responder:responding to #4952 from 10.200.187.77: 40956
2012-06-28 14:12:20,477 WARN Org.apache.hadoop.ipc.Server:IPC Server Responder, call Delete (/tmp/hive-hadoop/hive_ 2012-06-28_14-09-10_170_1995016130043493038/-mr-10057/e2239dfc-0947-4dc6-adc0-50e28f8e5c71, True) from 10.200.187.77:40956:output Error
2012-06-28 14:12:20,477 INFO Org.apache.hadoop.ipc.Server:IPC Server handler 289 on 8020 Caught:java.nio.channels.Close Dchannelexception
At Sun.nio.ch.SocketChannelImpl.ensureWriteOpen (socketchannelimpl.java:133)
At Sun.nio.ch.SocketChannelImpl.write (socketchannelimpl.java:325)
At Org.apache.hadoop.ipc.Server.channelWrite (server.java:1700)
At org.apache.hadoop.ipc.server.access$2000 (server.java:94)
At Org.apache.hadoop.ipc.server$responder.processresponse (server.java:741)
At Org.apache.hadoop.ipc.server$responder.dorespond (server.java:804)
At Org.apache.hadoop.ipc.server$handler.run (server.java:1416)

In the server category:

void Doread (Selectionkey key) throws Interruptedexception {

int count = 0;

Connection C = (Connection) key.attachment ();

if (c = = null) {

Return

}

C.setlastcontact (System.currenttimemillis ());

try {

Count = C.readandprocess ();

catch (Interruptedexception Ieo) {

Log.info (GetName () + ": Readandprocess caught Interruptedexception", IEO);

Throw Ieo;

catch (Exception e) {

Log.info (GetName () + ": Readandprocess threw exception" + E + ". Count of bytes read: "+ count, E);

Count = -1;//so That's (count < 0) is executed

}

if (Count < 0) {

if (log.isdebugenabled ())

Log.debug (GetName () + ": Disconnecting client" +

C + ". Number of active connections: "+

Numconnections);

CloseConnection (c);

c = null;

}

else {

C.setlastcontact (System.currenttimemillis ());

}

}

Careful study of the source call mode, found that remote RPC calls in the transmission of the method name, number of parameters, type, value and other information. However, when the server side reads, it still appears that count is less than 0.

After you intend to repair the source code, and then to test. Relevant results will be released in the near future.

After 3 weeks of continuous testing and tuning, the cause has been identified and will be sorted out later ........... ...

A summary of the combined use of Hadoop and hive (on a problem of stability)