Hadoop File Upload error Solution

Hadoop File Upload error Solution

It is said that the firewall and datanode are not properly started, but the check is normal. Later, we found a solution on a foreigner's website.

The/etc/security/limits. conf file is modified and uploaded successfully.

These Hadoop errors are all inexplicable. From this log, we can't see this problem. It seems that we still need to accumulate it on our own.

* Soft nofile 65536

* Hard nofile 65536

Hadoop dfs-put 1.txt/input/

The error log is as follows:

15/06/24 14:45:40 WARN util. NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

15/06/24 14:45:41 INFO hdfs. DFSClient: Exception in createBlockOutputStream

Java. io. EOFException: Premature EOF: no length prefix available

At org. apache. hadoop. hdfs. protocolPB. PBHelper. vintPrefixed (PBHelper. java: 2103)

At org. apache. hadoop. hdfs. DFSOutputStream $ DataStreamer. createBlockOutputStream (DFSOutputStream. java: 1380)

At org. apache. hadoop. hdfs. DFSOutputStream $ DataStreamer. nextBlockOutputStream (DFSOutputStream. java: 1302)

At org. apache. hadoop. hdfs. DFSOutputStream $ DataStreamer. run (DFSOutputStream. java: 536)

15/06/24 14:45:41 INFO hdfs. DFSClient: Abandoning BP-651950990-127.0.0.1-1435153229562: blk_rj3741839_1015

15/06/24 14:45:41 INFO hdfs. DFSClient: Excluding datanode 10.25.5.102: 50010

15/06/24 14:45:41 INFO hdfs. DFSClient: Exception in createBlockOutputStream

Java. io. EOFException: Premature EOF: no length prefix available

At org. apache. hadoop. hdfs. protocolPB. PBHelper. vintPrefixed (PBHelper. java: 2103)

At org. apache. hadoop. hdfs. DFSOutputStream $ DataStreamer. createBlockOutputStream (DFSOutputStream. java: 1380)

At org. apache. hadoop. hdfs. DFSOutputStream $ DataStreamer. nextBlockOutputStream (DFSOutputStream. java: 1302)

At org. apache. hadoop. hdfs. DFSOutputStream $ DataStreamer. run (DFSOutputStream. java: 536)

15/06/24 14:45:41 INFO hdfs. DFSClient: Abandoning BP-651950990-127.0.0.1-1435153229562: blk_rj3741840_1016

15/06/24 14:45:41 INFO hdfs. DFSClient: Excluding datanode 10.25.5.101: 50010

15/06/24 14:45:41 WARN hdfs. DFSClient: DataStreamer Exception

Org. apache. hadoop. ipc. remoteException (java. io. IOException): File/input/1.txt. _ COPYING _ cocould only be replicated to 0 nodes instead of minReplication (= 1 ). there are 2 datanode (s) running and 2 node (s) are excluded in this operation.

At org. apache. hadoop. hdfs. server. blockmanagement. BlockManager. chooseTarget (BlockManager. java: 1492)

At org. apache. hadoop. hdfs. server. namenode. FSNamesystem. getAdditionalBlock (FSNamesystem. java: 3027)

At org. apache. hadoop. hdfs. server. namenode. NameNodeRpcServer. addBlock (NameNodeRpcServer. java: 614)

At org. apache. hadoop. hdfs. server. namenode. AuthorizationProviderProxyClientProtocol. addBlock (AuthorizationProviderProxyClientProtocol. java: 188)

At org. apache. hadoop. hdfs. protocolPB. ClientNamenodeProtocolServerSideTranslatorPB. addBlock (ClientNamenodeProtocolServerSideTranslatorPB. java: 476)

At org. apache. hadoop. hdfs. protocol. proto. ClientNamenodeProtocolProtos $ ClientNamenodeProtocol $2. callBlockingMethod (ClientNamenodeProtocolProtos. java)

At org. apache. hadoop. ipc. ProtobufRpcEngine $ Server $ ProtoBufRpcInvoker. call (ProtobufRpcEngine. java: 587)

At org. apache. hadoop. ipc. RPC $ Server. call (RPC. java: 1026)

At org. apache. hadoop. ipc. Server $ Handler $ 1.run( Server. java: 2013)

At org. apache. hadoop. ipc. Server $ Handler $ 1.run( Server. java: 2009)

At java. security. AccessController. doPrivileged (Native Method)

At javax. security. auth. Subject. doAs (Subject. java: 422)

At org. apache. hadoop. security. UserGroupInformation. doAs (UserGroupInformation. java: 1614)

At org. apache. hadoop. ipc. Server $ Handler. Execute (Server. java: 2007)

 

At org. apache. hadoop. ipc. Client. call (Client. java: 1411)

At org. apache. hadoop. ipc. Client. call (Client. java: 1364)

At org. apache. hadoop. ipc. ProtobufRpcEngine $ Invoker. invoke (ProtobufRpcEngine. java: 206)

At com. sun. proxy. $ Proxy9.addBlock (Unknown Source)

At org. apache. hadoop. hdfs. protocolPB. ClientNamenodeProtocolTranslatorPB. addBlock (ClientNamenodeProtocolTranslatorPB. java: 391)

At sun. reflect. NativeMethodAccessorImpl. invoke0 (Native Method)

At sun. reflect. NativeMethodAccessorImpl. invoke (NativeMethodAccessorImpl. java: 62)

At sun. reflect. DelegatingMethodAccessorImpl. invoke (DelegatingMethodAccessorImpl. java: 43)

At java. lang. reflect. Method. invoke (Method. java: 497)

At org. apache. hadoop. io. retry. RetryInvocationHandler. invokeMethod (RetryInvocationHandler. java: 187)

At org. apache. hadoop. io. retry. RetryInvocationHandler. invoke (RetryInvocationHandler. java: 102)

At com. sun. proxy. $ Proxy10.addBlock (Unknown Source)

At org. apache. hadoop. hdfs. DFSOutputStream $ DataStreamer. locateFollowingBlock (DFSOutputStream. java: 1473)

At org. apache. hadoop. hdfs. DFSOutputStream $ DataStreamer. nextBlockOutputStream (DFSOutputStream. java: 1290)

At org. apache. hadoop. hdfs. DFSOutputStream $ DataStreamer. run (DFSOutputStream. java: 536)

Put: File/input/1.txt. _ COPYING _ cocould only be replicated to 0 nodes instead of minReplication (= 1 ). there are 2 datanode (s) running and 2 node (s) are excluded in this operation.

You may also like the following articles about Hadoop:

Tutorial on standalone/pseudo-distributed installation and configuration of Hadoop2.4.1 under Ubuntu14.04

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition