Storm on YARN installation deployment and sentiment

After installing storm on a single machine and successfully running WordCount, go to the next step in this week's work: Familiarize yourself with storm on yarn. A familiar first step is to install and deploy.

Existing environment: Three servers, HADOOP01/HADOOP02/HADOOP03, have installed the Hadoop version 2.2.0, have yarn environment and HDFS environment.

Required Software and configuration:

(1) Install Storm ZEROMQ and JZMQ, see more http://blog.csdn.net/flyhighzy/article/details/37672119

2 Storm-on-yarn Source Pack and maven (for compiling Storm-on-yarn)

3) Storm working environment

Specific details can be referred to Http://www.tuicool.com/articles/BFr2Yv

Some of the problems encountered:

1 when the storm to the yarn, the error "Can not find Storm home Entry", after the search, because the Storm folder naming problem, I downloaded the 0.9.2-incubating version, named Apache-storm-{version}, and when packaged as Storm.zip, I looked for the Storm-{version folder, which would cause you to not find it. So after renaming the Storm folder, you can pack it.

Reference: HTTPS://GITHUB.COM/YAHOO/STORM-YARN/ISSUES/68

2 when the first submission of storm to yarn, Only set Storm.zookeeper.servers and nimbus.host two options for the Storm.yaml configuration file, and then find that storm this application on yarn has been accept and not into the running state, so it may be that your own cluster is relatively small and memory Not enough, and the storm default container memory requirements are large, so yarn cannot allocate large enough container for it to cause storm to start running. The master.initial-num-supervisors and MASTER.CONTAINER.SIZE-MB two parameters were added to the Storm.yaml and then repackaged as Storm.zip to submit, and this was successfully run ^_^

3 JDK must use more than 1.7 version, otherwise there will be incompatible problems. This question in the Storm-yarn project's readme also wrote needs Jdk7, at that time did not pay attention to look, wasted some time, really is not worth.

Current Progress:

Storm can be submitted to the yarn environment, but can not submit storm tasks to the Storm-on-yarn application run, the current run WordCount will have the following error:

Exception in thread "main" java.lang.RuntimeException:org.apache.thrift7.transport.TTransportException: Java.net.ConnectException:Connection refused
At Backtype.storm.utils.NimbusClient.getConfiguredClient (nimbusclient.java:38)

In addition to open Storm-ui will also have connection refused error, the problem remains to be resolved.

Summary:

Through the installation and deployment, realized that storm is as an application will run for a long time on the yarn, Storm-yarn main implementation of two parts, client and applicationmaster,client for the submission and management of storm applications, AM is responsible for the initialization and logical control of the Nimbus, supervisor, and other servers within storm. So Jstorm-on-yarn can also take a similar structure.

=============================7.17 Update =========================================================

Yesterday's error viewing the log found that the actual error is:

Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

After Google found that zookeeper did not start, because the previous Hadoop cluster did not install a separate boot zookeeper. Start and then submit topology, which can run normally.

Next work:

Comparing the similarities and differences between Storm and Jstorm, the realization of design jstorm on yarn