AAS代碼運行-第11章-2

標籤：

hdfs dfs -ls /user/littlesuccess/AdvancedAnalysisWithSpark
hdfs dfs -mkdir /user/littlesuccess/AdvancedAnalysisWithSpark/ch11
hdfs dfs -put fish.py /user/littlesuccess/AdvancedAnalysisWithSpark/ch11

做好上述準備工作之後，就可以運行pyspark代碼了:

raw_data = sc.textFile(‘hdfs://172.31.25.243:8020/user/littlesuccess/AdvancedAnalysisWithSpark/ch11/fish.py‘)
data = (raw_data.filter(lambda x: x.startswith("#")).map(lambda x: map(float, x.split(‘,‘)))) 

                              data.take(5)

運行過程中發現了一個錯誤：

>>> data.take(5)Traceback (most recent call last):  File "<stdin>", line 1, in <module>  File "/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/spark/python/pyspark/rdd.py", line 1081, in take    totalParts = self._jrdd.partitions().size()  File "/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__  File "/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_valuepy4j.protocol.Py4JJavaError: An error occurred while calling o31.partitions.: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby    at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)    at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1713)    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1322)    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3974)    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:813)    at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getFileInfo(AuthorizationProviderProxyClientProtocol.java:502)    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:815)    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)    

發現原因在於我的叢集設定了NameNode HA,而我的指令碼中的hdfs用的是StandBy NameNode的地址，這個問題就解決了。

重新運行命令，又發現如下錯誤:

AAS代碼運行-第11章-2