Hadoop O & M Note: Snappy creates libhadoop. so, resulting in datanode Error

To solve the Bug mentioned in the previous article, we upgraded the online CDH5 to the latest CDH5.2.0. However, after the upgrade, datanode of some servers cannot be started normally and the error is as follows: 2014-11-9:9:54:52, 071WARNorg. apache. hadoop. hdfs. server. datanode. dataNode: Unexpectedexceptioninb

To solve the Bug mentioned in the previous article, we upgraded online CDH5 to the latest CDH5.2.0. However, after the upgrade, datanode of some servers cannot be started normally and the error is as follows: 19:54:52, 071 WARN org. apache. hadoop. hdfs. server. datanode. dataNode: Unexpected exception in B

To solve the Bug mentioned in the previous article, we upgraded the online CDH5 to the latest CDH5.2.0. However, after the upgrade, datanode of some servers cannot be started normally and the error is as follows:

2014-11-20 19:54:52,071 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in block pool Block pool 
 
   (Datanode Uuid unassigned) service to idc1-server1/10.100.1.100:8020com.google.common.util.concurrent.ExecutionError: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO.link0(Ljava/lang/String;Ljava/lang/String;)Vat com.google.common.util.concurrent.Futures.wrapAndThrowExceptionOrError(Futures.java:1126)at com.google.common.util.concurrent.Futures.get(Futures.java:1048)at org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocks(DataStorage.java:870)at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.linkAllBlocks (BlockPoolSliceStorage.java:570)at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doUpgrade (BlockPoolSliceStorage.java:379)at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.doTransition (BlockPoolSliceStorage.java:313)at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead (BlockPoolSliceStorage.java:187)at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead (DataStorage.java:309)at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1109)at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1080)at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo (BPOfferService.java:320)at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake (BPServiceActor.java:220)at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824)at java.lang.Thread.run(Thread.java:744)Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO.link0 (Ljava/lang/String;Ljava/lang/String;)Vat org.apache.hadoop.io.nativeio.NativeIO.link0(Native Method)at org.apache.hadoop.io.nativeio.NativeIO.link(NativeIO.java:838)at org.apache.hadoop.hdfs.server.datanode.DataStorage$2.call(DataStorage.java:862)at org.apache.hadoop.hdfs.server.datanode.DataStorage$2.call(DataStorage.java:855)at java.util.concurrent.FutureTask.run(FutureTask.java:262)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)... 1 more2014-11-20 19:54:52,073 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool 
  
    (Datanode Uuid unassigned) service to idc1-server1/10.100.1.100:8020
  
 

However, Google does not find any matching information. The only information with vertices and edges is caused by the lack of lib on the Windows platform.
In our environment, only some servers have the above problems. After comparing all Hadoop-related software packages, we cannot find any difference, this has brought us a lot of interference in our analysis.

Finally, we try to use strace to track the datanode process.
Yum install strace
Strace-f-F-o/tmp/strace.output.txt/etc/init. d/hadoop-hdfs-datanode start
Lsof | grep libhadoop. so

java 18527 hdfs mem REG 253,0 122832 270200 /usr/java/jdk1.7.0_45/jre/lib/amd64/libhadoop.so

It is found that it reads a lib file:/usr/java/jdk1.7.0 _ 45/jre/lib/amd64/libhadoop. so, while the datanode process of other normal servers is read/usr/lib/hadoop/lib/native/libhadoop. so.
It is verified that/usr/java/jdk1.7.0 _ 45/jre/lib/amd64/libhadoop. so is created when the Snappy software package is installed. After it is removed, datanode is started properly.

It seems that although datanode specifies-Djava. library. path =/usr/lib/hadoop/lib/native at startup, the priority of lib loading in jre is higher.

Original article address: The datanode error is reported when libhadoop. so is created by Snappy in the Hadoop O & M notes. Thanks for sharing the original article.