Storm on yarn installation is submitted to yarn failure failed

Recently deploying storm on Yarn , deploying reference articles

http://www.tuicool.com/articles/BFr2Yv
http://blog.csdn.net/jiushuai/article/details/18729367

After installing zookeeper, configure Storm and Storm on yarn, start zookeeper, where zookeeper port is 2181,
Then compile the project through the MVN package, find that an error occurs, and then recompile with MVN packet-dskiptests, skipping test
Then submit the storm task to yarn, Storm-yarn launch <path to your Storm.yaml file>

Review localhost:8088 after submission , find task failed, see error message found below

Application application_1411179375629_0005 failed2Times due to AM Container forAppattempt_1411179375629_0005_000002 exited with ExitCode:1Due to:exception fromcontainer-Launch:org.apache.hadoop.util.shell$exitcodeexception:org.apache.hadoop.util.shell$exitcodeexception:at Org.apache.hadoop.util.Shell.runCommand (Shell.java:505) at Org.apache.hadoop.util.Shell.run (Shell.java:418) at Org.apache.hadoop.util.shell$shellcommandexecutor.execute (Shell.java:650) at Org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer ( Defaultcontainerexecutor.java:195) at Org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call ( Containerlaunch.java:283) at Org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call ( Containerlaunch.java: -) at Java.util.concurrent.FutureTask.run (Futuretask.java:262) at Java.util.concurrent.ThreadPoolExecutor.runWorker (Threadpoolexecutor.java:1145) at Java.util.concurrent.threadpoolexecutor$worker.run (Threadpoolexecutor.java:615) at Java.lang.Thread.run (Thread.java:745) Container exited with a non-zero Exit code1. Failing ThisAttempt. Failing the application.

Check the log information to find specific errors as follows

 -/ the/ +  -: -: -INFO yarn. Masterserver:starting Master Thrift Server -/ the/ +  -: -: -ERROR Auth. Thriftserver:thriftserver isBeing stopped due to:org.apache.thrift7.transport.TTransportException:Could not create serversocket on address0.0.0.0/0.0.0.0:9000. Org.apache.thrift7.transport.TTransportException:Could not create serversocket on address0.0.0.0/0.0.0.0:9000. At Org.apache.thrift7.transport.TNonblockingServerSocket.<init> (Tnonblockingserversocket.java: the) at Org.apache.thrift7.transport.TNonblockingServerSocket.<init> (Tnonblockingserversocket.java: the) at Org.apache.thrift7.transport.TNonblockingServerSocket.<init> (Tnonblockingserversocket.java: A) at Backtype.storm.security.auth.SimpleTransportPlugin.getServer (Simpletransportplugin.java: -) at Backtype.storm.security.auth.ThriftServer.serve (Thriftserver.java: the) at Com.yahoo.storm.yarn.MasterServer.main (Masterserver.java:175)

An error occurred while booting Master Thrift server, the error is a port error because port 9000 is already being monitored by HDFS and therefore cannot be created ServerSocket

Thanks to Google, the solution is to modify the port number, the post is as follows

Https://groups.google.com/forum/#!topic/storm-yarn/A1ds1M6qmN8

Modify the Storm-yarn-master/src/main/resources/master_defaults.yaml, change the master.thrift.port to one of the other appropriate values, and I'll change it to 9001.

Then compile the project, resubmit, then found that the task is not failed, but access to localhost:7070, cannot access, after viewing the log found that the error is Nimbus did not start successfully, the error is as follows:

 the/ -/ the Geneva: $: .ERROR yarn. masterserver:unhandled ErrorinchAM:org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException:Invalid resource request, requestedVirtualCores <0, or requestedVirtualCores > Max configured, requestedvirtualcores= the, maxvirtualcores=8At Org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest ( Schedulerutils.java:213) at Org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests (Rmserverutils.java : the) at Org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate ( Applicationmasterservice.java:502)    ... ... ... ... ... ...At com.yahoo.storm.yarn.masterserver$1. Run (Masterserver.java: the) caused By:org.apache.hadoop.ipc.RemoteException ( org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException): Invalid resource request, requestedVirtualCores <0, or requestedVirtualCores > Max configured, requestedvirtualcores= the, maxvirtualcores=8At Org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest ( Schedulerutils.java:213) at Org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests (Rmserverutils.java : the)

    ... ... ... ... ... ...

At Com.sun.proxy. $Proxy 7.allocate (Unknown Source) at Org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate ( Applicationmasterprotocolpbclientimpl.java: the) ... 9 More the/ -/ the Geneva: $: .INFO yarn. Stormmasterserverhandler:stopping Supervisors ... the/ -/ the Geneva: $: .INFO yarn. Stormmasterserverhandler:stopping UI ... the/ -/ the Geneva: $: .INFO yarn. Stormmasterserverhandler:stopping Nimbus ...

This means that the requested virtual cores exceeds the maximum limit maxvirtualcores,google find post

Hadoop-why cannot more than cores is requested from YARN to run a job? -Stack Overflow
Http://stackoverflow.com/questions/29780401/why-cannot-more-than-32-cores-be-requested-from-yarn-to-run-a-job

So modify the Yarn-site.xml as follows

<property>  <name>yarn.nodemanager.resource.cpu-vcores</name>  <value>8 </value></property>

Restart yarn, then submit, Success!!!

If you find the following error

 2015 -07 - 15  03 :  : 28  o.a.z.clientcnxn [INFO] Opening socket connection to server mmc/< Span style= "color: #800080;" >192.168 . 1.200 : 2181  2015 -07 - 15  03 :  : 28  o.a.z.clientcnxn [WARN] Session  for  server null  , unexpected error, closing socket connection and attempting reconnectjava.net.ConnectException: Deny connection  

It is possible that the zookeeper is not started and can be started

Always pay attention to your firewall is not closed, some unknown reason is because the firewall did not shut down caused!

It took three or four days to solve the problem, during a lot of detours, because just started to contact Hadoop storm on yarn, so sometimes out of error do not know how to check the wrong, the beginning is only their own estimate of what has been the problem, and then change to run again, the result or not, Later learned to check the error log, under the masterhost:8088/logs/is all the error logs, and then find the related error log troubleshooting

Acknowledgements: Google

Spit a trough of the evil net G-Audit W-check ￥ system # degrees, I'm a damn I'm a damn I'm a hell of a. The Internet on the seclusion!!!

Storm on yarn installation is submitted to yarn failure failed