Storm on Yarn Deployment __storm

1. Introduction to the environment 1.1 node and service mapping relationship

IP	Host	Service
192.168.40.132	Master	Namenode, NodeManager, Datanode, zookeeper
192.168.40.133	Slave1	Resurcemanager, NodeManager, Datanode, zookeeper
192.168.40.134	Slave2	NodeManager, Datanode, zookeeper

1.2 JDK version

[Root@slave2 ~]# java-version
java Version "1.7.0_79"
Java (TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot (TM) 64-bit Server VM (build 24.79-b02, Mixed mode)

1.3 Hadoop version

[Root@slave2 ~]# Hadoop version
Hadoop 2.6.0
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git- R E3496499ECB8D220FBA99DC5ED4C99C8F9E33BB1
Compiled by Jenkins in 2014-11-13t21:10z
Compiled with Protoc 2.5.0 From
source with checksum 18e43357c8f927c0695f1e9522859d6a
This command is run Using/application/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar

1.4 Each node process JPS

2. Install maven 2.1 Download, extract maven

wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

TAR-XZVF apache-maven-3.3.9-bin.tar.gz

ln-s Apache-maven-3.3.9/maven

2.2 Configuring MAVEN environment variables

Export Maven_home=/application/maven
export path=.: $MAVEN _home/bin: $JAVA _home/bin: $HADOOP _home/bin: $HADOOP _ Home/sbin: $PATH

source/etc/profile

2.3 Authentication, view MAVEN version number

Mvn-v
Apache maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5 2015-11-11t00:41:47+08:00)
maven Home:/ Application/maven
java version:1.7.0_79, vendor:oracle Corporation
java home:/application/jdk1.7.0_79/jre
Default Locale:en_us, platform Encoding:utf-8
OS name: "Linux", Version: "2.6.32-504.el6.x86_64", Arch: "AMD64", Family: "Unix"

3. Install Storm-yarn 3.1 Download Storm-yarn

wget Https://github.com/yahoo/storm-yarn/archive/master.zip

Rename the downloaded compressed package to Storm-yarn.zip

MV Master.zip Storm-yarn.zip

Extract

Unzip Storm-yarn.zip

3.2 Modification Pom.xml

Modify Storm-yarn-master/pom.xml According to the version number of Hadoop

<properties>
    <storm.version>0.9.6</storm.version>
    
 
Because the version of Hadoop in my environment is 2.6.0, there is no storm (and no installation), so pom.xml is not modified. 3.3 Compiling package Storm-yarn-master

MVN package-dskiptests
 
Package successful

[INFO]---maven-jar-plugin:2.4:jar (default-jar) @ storm-yarn---
[info]--------------------------------------- -------------------------------
[INFO] Build SUCCESS
[INFO]--------------------------------------------- -------------------------
[INFO] Total time:08:41 min
[info] finished at:2016-08-26t10:16:27+08:00
[ Info] Final memory:14m/34m
[INFO]----------------------------------------------------------------------
 
3.4 Extract Lib/storm.zip and move the unpacked directory 
Decompression Storm-yarn-master/lib/storm.zip

Unzip Storm-yarn-master/lib/storm.zip 

[root@master lib]# ls
storm-0.9.0-wip21  storm-0.9.0-wip21.zip  Storm-starter-0.0.1-snapshot.jar  Storm.zip
 
Move the extracted storm-0.9.0-wip21 to the Storm-yarn-master sibling directory

[Root@master lib]mv storm-0.9.0-wip21. /..
[Root@master lib]# CD ... /..
[Root@master application]# ls
maven   storm-yarn.zip  apache-maven-3.3.9  tools   Hadoop  Zookeeper   hadoop-2.6.0    java    storm-0.9.0-wip21   zookeeper-3.3.6 jdk1.7.0_79 storm-yarn-master   
 
3.5 Configuring environment Variables 
Export path=.:/ Application/storm-0.9.0-wip21/bin:/application/storm-yarn-master/bin: $MAVEN _home/bin:: $JAVA _home/bin: $HADOOP _ Home/bin: $HADOOP _home/sbin: $PATH
 
Use Source/etc/profile to make it effective 3.6 edit Storm-0.9.0-wip21/conf/storm.yaml

Modify Zookeeper Address

Storm.zookeeper.servers:
     -"Master"-"
     slave1"
     -"slave2"
 
3.7 Compressed Storm.zip and uploaded onto HDFs . 
Compressed STORM-0.9.0-WIP21

 
Upload the compressed file storm.zip to the directory on HDFs/lib/storm/0.9.0-wip21/

Hadoop fs-mkdir-p/lib/storm/0.9.0-wip21
hdfs dfs-put storm.zip/lib/storm/0.9.0-wip21/
HDFs m/0.9.0-wip21/
Found 1 Items
-rw-r--r--   3 root supergroup   17141097 2016-08-26 10:27/lib/storm/0.9.0- Wip21/storm.zip
 
4. Run Storm-yarn 
4.1 Start Storm-yarn 
[Root@master application]# storm-yarn launch Storm-0.9.0-wip21/conf/storm.yaml slf4j:class path contains multiple SLF4J
Bindings. Slf4j:found Binding in [jar:file:/application/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/ Slf4j/impl/staticloggerbinder.class] Slf4j:found binding in [jar:file:/application/storm-0.9.0-wip21/lib/ Logback-classic-1.0.6.jar!/org/slf4j/impl/staticloggerbinder.class] Slf4j:see http://www.slf4j.org/codes.html#
Multiple_bindings for a explanation. Slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/08/26 10:28:28 INFO client. Rmproxy:connecting to ResourceManager at slave1/192.168.40.133:8032 16/08/26 10:28:29 WARN. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable 16 /08/26 10:28:30 INFO yarn. Stormonyarn:copy App Master jar from the local filesystem and add to local environment 16/08/26 10:28:32 INFO yarn. Stormonyarn:set the environment for the appLication Master 16/08/26 10:28:32 INFO yarn. Stormonyarn:yarn CLASSPATH COMMAND = [[YARN, CLASSPATH]] 16/08/26 10:28:32 INFO YARN. stormonyarn:using java_home = [/application/java] 16/08/26 10:28:32 INFO yarn. Stormonyarn:setting up App Master Command:[/application/java/bin/java,-dstorm.home=./storm/storm-0.9.0-wip21/,- Dlogfile.name=<log_dir>/master.log, Com.yahoo.storm.yarn.MasterServer, 1><log_dir>/stdout, 2> <log_dir>/stderr] 16/08/26 10:28:32 INFO impl. yarnclientimpl:submitted Application application_1472175545791_0002 application_1472175545791_0002
 
View the appplication operation of yarn through http://slave2:8088

4.2 Get Nimbus Address

Get Storm's property information and save it to/root/.storm/storm.yaml

[Root@master application]# mkdir-p/root/.storm/[root@master application]# Storm-yarn-Getstormconfig-appid
Application_1472175545791_0002-output ~/.storm/storm.yaml slf4j:class path contains multiple SLF4J. Slf4j:found Binding in [jar:file:/application/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/ Slf4j/impl/staticloggerbinder.class] Slf4j:found binding in [jar:file:/application/storm-0.9.0-wip21/lib/ Logback-classic-1.0.6.jar!/org/slf4j/impl/staticloggerbinder.class] Slf4j:see http://www.slf4j.org/codes.html#
Multiple_bindings for a explanation. Slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/08/26 10:30:13 INFO client. Rmproxy:connecting to ResourceManager at slave1/192.168.40.133:8032 16/08/26 10:30:13 WARN. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable 16 /08/26 10:30:14 INFO yarn. Stormonyarn:application for application_1472175545791_0002 : slave2:9000 16/08/26 10:30:14 INFO yarn. Stormonyarn:attaching to slave2:9000 to talk to app master application_1472175545791_0002 {10:35:18}16/08/26 10:30:14 INF O yarn. StormMasterCommand:storm.yaml downloaded Into/root/.storm/storm.yaml
 
get Nimbus address based on/root/.storm/storm.yaml 
[Root@master. storm]# Cat ~/.storm/storm.yaml | grep nimbus.host
nimbus.host:192.168.40.134
 
4.3 access to storm UI via Nimbus 
Accessing the Storm UI by accessing http://192.168.40.134:7070

4.4 Submitting Tasks WordCount

[root@master storm-yarn-master]# Storm Jar Lib/storm-starter-0.0.1-snapshot.jar  Storm.starter.WordCountTopology wordcounttopology-c nimbus.host=192.168.40.134 561 [main] INFO Backtype.storm.stormsubmitter-jar not uploaded to master yet.
Submitting Jar ... 573 [main] INFO backtype.storm.stormsubmitter-uploading topology jar Lib/storm-starter-0.0.1-snapshot.jar to assigned  Location:storm-local/nimbus/inbox/stormjar-12a737f5-82e9-43a6-b422-c1740cc36635.jar 596 [main] INFO backtype.storm.stormsubmitter-successfully uploaded topology Jar to assigned location:storm-local/nimbus/inbox/ Stormjar-12a737f5-82e9-43a6-b422-c1740cc36635.jar 596 [main] INFO backtype.storm.stormsubmitter-submitting topology Wordcounttopology in Distributed mode with conf {"Nimbus.host": "192.168.40.134", "Topology.workers": 3, "Topology.debug ': true} 948 [main] INFO backtype.storm.stormsubmitter-finished submitting Topology:wordcounttopology 
 
View Storm UI 
View the operation of the topology WordCount

View each node JPS

5. Close 5.1 Close Storm submitted tasks

[Root@master ~]# storm kill Wordcounttopology
2277 [main] INFO  backtype.storm.thrift-connecting to Nimbus at 192 .168.40.134:6627
2387 [main] INFO  backtype.storm.command.kill-topology-killed topology:wordcounttopology
 
5.2 Off Storm on yarn

[Root@master ~]# yarn application-kill application_1472175545791_0002 killing application Application_
1472175545791_0002