A bug has gone through the HBase heap memory small causes Regionserver to hang up frequently

The environment is as follows:
Centos6.5
Apache Hadoop2.7.1
Apache Hbase0.98.12
Apache Zookeeper3.4.6
JDK1.7
Ant1.9.5
Maven3.0.5

Recently in the compression of HBase, Hadoop installed lzo and snappy, inserting 50 text data, each piece of data about 4M, to see their compression ratio comparison,
Then in the process of testing, found that the Java client to scan the 50 data, regionserver frequent downtime to see HBase log found no obvious exception, view Datanode log found the following exception:

Java code

2. Java.io.IOException:Premature EOF from InputStream
4. At org.apache.hadoop.io.IOUtils.readFully (Ioutils.java:201)
6. At org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully (Packetreceiver.java:213)
8. At Org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead (Packetreceiver.java:134)
10. At Org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket (Packetreceiver.java:109)
12. At Org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket (Blockreceiver.java:472)
14. At Org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock (Blockreceiver.java:849)
16. At Org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock (Dataxceiver.java:804)
18. At Org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock (Receiver.java:137)
20. At Org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp (Receiver.java:)
22. At Org.apache.hadoop.hdfs.server.datanode.DataXceiver.run (Dataxceiver.java:251)
24. At Java.lang.Thread.run (Thread.java:745)

Java.io.IOException:Premature EOF from InputStream at org.apache.hadoop.io.IOUtils.readFully (ioutils.java:201) At org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully (packetreceiver.java:213) at Org.apac He.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead (packetreceiver.java:134) at Org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket (packetreceiver.java:109) at Org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket (blockreceiver.java:472) at Org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock (blockreceiver.java:849) at Org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock (dataxceiver.java:804) at Org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock (receiver.java:137) at Org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp (receiver.java:74) at Org.apache.hadoop.hdfs.server.datanode.DataXceiver.run (dataxceiver. java:251) at Java.lang.Thread.run (thread.java:745) 

As below, OK, out of the ordinary, take this exception Google search results, found that there is no clear answer, most of the link timeout, or the handle is full, resulting in link interruption and so on, and then according to these answers, changed a number of configurations, found still not effective, this leads me to feel very strange, Come to a wrong conclusion, HBase does not support a variety of compression types coexist tables, and then I removed the other types used to compress the test table, test again, found that the problem is still, this again makes me very surprised, is not the environment problem? Because I can't think of a possible problem, and then test it on a native virtual machine,
The virtual machine environment, because it is self-used, so JDK version is 1.8 and the CentOS version is 7,hbase,hadoop,zookeeper version is consistent,
After the construction, continue to test, found that the problem is still, this is more confusing, see out of the non-environmental problems, but this time there is a new clue, because the use of JDK8, in HBase log inside found a large number of full GC log, meaning that the memory is seriously insufficient, Lead to garbage collection time appeared 4, 5 seconds, this is a bit of a clue, HBase is a memory-eating thing, memory to give less, it is possible to cause regionserver hang up, so I look at the heap memory allocation of hbase, found that the default 1G, this really has a big relationship with this, 50 data for storage 200M, if each scan, HBase will cache it in the cache, the second time to continue to scan different compression types of tables, will cause memory bloat, and then throw, regionserver down, and the given exception hint is not very clear, So it is difficult to locate the problem, know the probable cause, and then put HBase heap memory to 4G, and distributed to all nodes, start again, with the Java client, scan the full table test, this time is very stable, regionserver did not appear once again hanging off the situation.









Finally, a conclusion is given for the test compression: A total of 4 compression comparisons are measured, the original data 200M
(1) Without compression space 128.1 M
(2) GZ compression accounted for 920.3 K
(3) Snappy compression accounted for 13.2M
(4) Lzo compression accounted for 8M

The above can be seen, GZ compression ratio is highest, lzo second, snappy third, of course, different compression for the use of business scenarios, here can not be simple
Summary must use what, this inside snappy and lzo at present most Internet companies use more, so we can according to the specific business, to choose the appropriate compression scheme.

A bug has gone through the HBase heap memory small causes Regionserver to hang up frequently