Storm on yarn Installation __storm

1: Download

[Jifeng@feng02 storm]$ wget https://github.com/yahoo/storm-yarn/archive/master.zip
--2015-03-08 21:07:24--  Https://github.com/yahoo/storm-yarn/archive/master.zip
is parsing host github.com ... 192.30.252.130
is connecting github.com|192.30.252.130|:443 ... connected.
HTTP request issued, waiting for response ... 302 Found
Position: https://codeload.github.com/yahoo/storm-yarn/zip/master [follow to new URL]
--2015-03-08 21:07:27--  Https://codeload.github.com/yahoo/storm-yarn/zip/master
is parsing host codeload.github.com ... 192.30.252.145
is connecting codeload.github.com|192.30.252.145|:443 ... connected.
HTTP request issued, waiting for response ... OK
Length: Not specified [Application/zip]
is being saved to: "Master"

    [                           <=>                                                         ] 17,346,738   288k/s in   2m 0s   

2015-03-08 21:09:30 (kb/s)-"Master" saved [17346738]

2: Decompression

[Jifeng@feng02 storm]$ mv Master Storm-yarn.zip [jifeng@feng02 storm]$ unzip Storm-yarn.zip archive:storm-yarn.zip e570e 9665bac68396db09a9711875948cc63540e Creating:storm-yarn-master/inflating:storm-yarn-master/cla.pdf Inflating: Storm-yarn-master/license.txt inflating:storm-yarn-master/readme.md creating:storm-yarn-master/bin/inflating : Storm-yarn-master/bin/read-link Inflating:storm-yarn-master/bin/storm-yarn inflating:storm-yarn-master/create- tarball.sh Creating:storm-yarn-master/lib/inflating:storm-yarn-master/lib/storm-0.9.0-wip21.zip inflating:s Torm-yarn-master/lib/storm-starter-0.0.1-snapshot.jar Linking:storm-yarn-master/lib/storm.zip-> storm-0.9.0-w Ip21.zip Inflating:storm-yarn-master/pom.xml creating:storm-yarn-master/src/creating:storm-yarn-master/src/ Main/inflating:storm-yarn-master/src/main/genthrift.sh creating:storm-yarn-master/src/main/java/creating:st orm-yarn-master/src/main/Java/com/creating:storm-yarn-master/src/main/java/com/yahoo/creating:storm-yarn-master/src/main/java/com/yahoo /storm/creating:storm-yarn-master/src/main/java/com/yahoo/storm/yarn/inflating:storm-yarn-master/src/main/java /com/yahoo/storm/yarn/client.java Inflating:storm-yarn-master/src/main/java/com/yahoo/storm/yarn/config.java inf Lating:storm-yarn-master/src/main/java/com/yahoo/storm/yarn/launchcommand.java inflating:storm-yarn-master/src/ Main/java/com/yahoo/storm/yarn/masterclient.java Inflating:storm-yarn-master/src/main/java/com/yahoo/storm/yarn /masterserver.java Inflating:storm-yarn-master/src/main/java/com/yahoo/storm/yarn/stormamrmclient.java Inflating : Storm-yarn-master/src/main/java/com/yahoo/storm/yarn/stormmastercommand.java inflating:storm-yarn-master/src/ Main/java/com/yahoo/storm/yarn/stormmasterserverhandler.java inflating:storm-yarn-master/src/main/java/com/ Yahoo/storm/yarn/stormonyarn.java Inflating:storm-yarN-master/src/main/java/com/yahoo/storm/yarn/util.java Inflating:storm-yarn-master/src/main/java/com/yahoo/storm /yarn/version.java Inflating:storm-yarn-master/src/main/java/com/yahoo/storm/yarn/versioncommand.java Creating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/generated/inflating:storm-yarn-master/src/main/java/com/ Yahoo/storm/yarn/generated/stormmaster.java creating:storm-yarn-master/src/main/resources/inflating:storm-yarn- Master/src/main/resources/logback.xml Inflating:storm-yarn-master/src/main/resources/master_defaults.yaml Inflat Ing:storm-yarn-master/src/main/storm_master.thrift Creating:storm-yarn-master/src/test/creating:storm-yarn-mas ter/src/test/java/creating:storm-yarn-master/src/test/java/com/creating:storm-yarn-master/src/test/java/com/ yahoo/creating:storm-yarn-master/src/test/java/com/yahoo/storm/creating:storm-yarn-master/src/test/java/com/ yahoo/storm/yarn/inflating:storm-yarn-master/Src/test/java/com/yahoo/storm/yarn/embeddedzkserver.java inflating:storm-yarn-master/src/test/java/com/yahoo/ Storm/yarn/testconfig.java Inflating:storm-yarn-master/src/test/java/com/yahoo/storm/yarn/testintegration.java I Nflating:storm-yarn-master/src/test/java/com/yahoo/storm/yarn/teststormcluster.java Inflating:storm-yarn-master
  /src/test/java/com/yahoo/storm/yarn/teststormmaster.java creating:storm-yarn-master/src/test/resources/ Inflating:storm-yarn-master/src/test/resources/log4j.properties inflating:storm-yarn-master/src/test/resources/ Logback-test.xml Finishing deferred symbolic links:storm-yarn-master/lib/storm.zip-> Storm-0.9.0-wip21.zip [Jifen G@FENG02 storm]$ ls storm-yarn-master storm-yarn.zip [jifeng@feng02 storm]$ cd storm-yarn-master/[jifeng@feng02 storm-y arn-master]$ ls bin cla.pdf create-tarball.sh lib LICENSE.txt pom.xml readme.md src

3: Compiling

[Jifeng@feng02 storm-yarn-master]$ mvn package-dskiptests [INFO] scanning for projects ...                                                                         
[INFO] [INFO]------------------------------------------------------------------------[INFO] Building Storm-yarn 1.0- Alpha [INFO]------------------------------------------------------------------------downloading:http:// Repo1.maven.org/maven2/storm/storm-core/0.9.0-wip21/storm-core-0.9.0-wip21.jar downloading:http:// Repo1.maven.org/maven2/org/clojure/clojure/1.4.0/clojure-1.4.0.jar downloading:http://repo1.maven.org/maven2/ Storm/libthrift7/0.7.0-2/libthrift7-0.7.0-2.jar downloading:http://repo1.maven.org/maven2/clj-time/clj-time/
0.4.1/clj-time-0.4.1.jar Downloading:http://repo1.maven.org/maven2/backtype/jzmq/2.1.0/jzmq-2.1.0.jar Downloading:http://repo1.maven.org/maven2/compojure/compojure/1.1.3/compojure-1.1.3.jar downloading:http:// Repo1.maven.org/maven2/clout/clout/1.0.1/clout-1.0.1.jar DOWNLOADING:HTTP://REPO1. Maven.org/maven2/ring/ring-core/1.1.5/ring-core-1.1.5.jar downloading:http://repo1.maven.org/maven2/hiccup/ Hiccup/0.3.6/hiccup-0.3.6.jar downloading:http://repo1.maven.org/maven2/ring/ring-devel/0.3.11/ Ring-devel-0.3.11.jar downloading:http://repo1.maven.org/maven2/clj-stacktrace/clj-stacktrace/0.2.2/ Clj-stacktrace-0.2.2.jar downloading:http://repo1.maven.org/maven2/ring/ring-jetty-adapter/0.3.11/ Ring-jetty-adapter-0.3.11.jar downloading:http://repo1.maven.org/maven2/ring/ring-servlet/0.3.11/
Ring-servlet-0.3.11.jar Downloading:http://repo1.maven.org/maven2/storm/carbonite/1.5.0/carbonite-1.5.0.jar Downloading:http://repo1.maven.org/maven2/storm/tools.cli/0.2.2/tools.cli-0.2.2.jar downloading:http:// Repo1.maven.org/maven2/storm/jgrapht/0.8.3/jgrapht-0.8.3.jar downloading:http://repo1.maven.org/maven2/com/ Google/guava/guava/13.0/guava-13.0.jar downloading:http://repo1.maven.org/maven2/storm/storm-netty/0.9.0-wip21/ Storm-netty-0.9.0-wip21.jar downloading:http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.1.0-beta/ Hadoop-yarn-server-nodemanager-2.1.0-beta.jar downloading:http://repo1.maven.org/maven2/org/apache/hadoop/ Hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar downloading:http://repo1.maven.org/maven2/net/java/dev/ Jets3t/jets3t/0.6.1/jets3t-0.6.1.jar downloaded:http://repo1.maven.org/maven2/org/apache/hadoop/
Hadoop-yarn-server-nodemanager/2.1.0-beta/hadoop-yarn-server-nodemanager-2.1.0-beta.jar (452 KB at 46.8 KB/sec) Downloading:http://repo1.maven.org/maven2/org/apache/avro/avro/1.5.3/avro-1.5.3.jar downloaded:http:// Repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar (315 KB at 25.4 KB/sec) downloading:http:// Repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar downloaded:http:// Repo1.maven.org/maven2/org/apache/avro/avro/1.5.3/avro-1.5.3.jar (258 KB at 18.0 kb/sec) downloading:http:// Repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-aUth-2.1.0-beta.jar downloaded:http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.1.0-beta/ Hadoop-auth-2.1.0-beta.jar (KB at 2.9 kb/sec) downloading:http://repo1.maven.org/maven2/org/apache/commons/ Commons-compress/1.4/commons-compress-1.4.jar downloaded:http://repo1.maven.org/maven2/org/apache/commons/ Commons-compress/1.4/commons-compress-1.4.jar (234 KB at 9.4 kb/sec) downloading:http://repo1.maven.org/maven2/org/ Apache/hadoop/hadoop-mapreduce-client-core/2.1.0-beta/hadoop-mapreduce-client-core-2.1.0-beta.jar Downloaded: Http://repo1.maven.org/maven2/com/google/guava/guava/13.0/guava-13.0.jar (1847 KB at 26.7 kb/sec) downloading:http:/
/repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.1.0-beta/hadoop-yarn-common-2.1.0-beta.jar downloaded:http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.1.0-beta/ Hadoop-mapreduce-client-core-2.1.0-beta.jar (1341 KB at 15.1 kb/sec) downloading:http://repo1.maven.org/maven2/org/ apache/hadoop/hAdoop-yarn-client/2.1.0-beta/hadoop-yarn-client-2.1.0-beta.jar downloaded:http://repo1.maven.org/maven2/org/ Xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar (973 KB at 7.8 kb/sec) downloading:http://repo1.maven.org/ Maven2/org/apache/hadoop/hadoop-yarn-api/2.1.0-beta/hadoop-yarn-api-2.1.0-beta.jar downloaded:http:// Repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.1.0-beta/hadoop-yarn-common-2.1.0-beta.jar (1264 KB at 8.3 kb/sec) downloading:http://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.3.3/ Zookeeper-3.3.3.jar

Downloading:https://clojars.org/repo/storm/storm-netty/0.9.0-wip21/storm-netty-0.9.0-wip21.jar Downloaded:https ://clojars.org/repo/storm/carbonite/1.5.0/carbonite-1.5.0.jar (MB KB at 8.9 kb/sec) downloaded:https://clojars.org/ Repo/storm/libthrift7/0.7.0-2/libthrift7-0.7.0-2.jar (296 KB at 33.4 kb/sec) downloaded:https://clojars.org/repo/ Storm/storm-netty/0.9.0-wip21/storm-netty-0.9.0-wip21.jar (KB at 4.7 kb/sec) downloaded:https://clojars.org/repo/ Storm/jgrapht/0.8.3/jgrapht-0.8.3.jar (KB at 18.9 kb/sec) downloaded:https://clojars.org/repo/storm/storm-core/ 0.9.0-wip21/storm-core-0.9.0-wip21.jar (5197 KB at 19.1 kb/sec) [INFO] [INFO]---maven-resources-plugin:2.6:resources ( Default-resources @ storm-yarn---[WARNING] Using platform encoding (UTF-8 actually) to copy filtered, i.e. bu
ILD is platform dependent! [INFO]  Copying 2 Resources [INFO] [INFO]---maven-compiler-plugin:3.0:compile (default-compile) @ storm-yarn---[info] Changes Detected-recompiling THe module! [WARNING]
File encoding has not been set, using platform encoding UTF-8, i.e. builds is platform dependent! [INFO] compiling source files to/home/jifeng/hadoop/storm/storm-yarn-master/target/classes [info] [INFO]--- Maven-resources-plugin:2.6:testresources (default-testresources) @ storm-yarn---[WARNING] Using platform encoding (
UTF-8 actually) to copy filtered, i.e. builds is platform dependent! [INFO]  Copying 2 Resources [INFO] [INFO]---maven-compiler-plugin:3.0:testcompile (default-testcompile) @ storm-yarn---[info]
Changes detected-recompiling the module! [WARNING]
File encoding has not been set, using platform encoding UTF-8, i.e. builds is platform dependent! [INFO] Compiling 5 source files to/home/jifeng/hadoop/storm/storm-yarn-master/target/test-classes [info] [INFO]--- Maven-jar-plugin:2.4:jar (default) @ Storm-yarn---[INFO] building jar:/home/jifeng/hadoop/storm/storm-yarn-master/ Target/storm-yarn-1.0-alpha.jar [INFO] [INFO]---maven-sUrefire-plugin:2.10:test (default-test) @ Storm-yarn---downloading:https://repo.maven.apache.org/maven2/org/ Apache/maven/surefire/surefire-booter/2.10/surefire-booter-2.10.pom downloaded:https://repo.maven.apache.org/ Maven2/org/apache/maven/surefire/surefire-booter/2.10/surefire-booter-2.10.pom (3 KB at 0.8 KB/sec) Downloading:
Https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.10/surefire-api-2.10.pom downloaded:https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.10/ Surefire-api-2.10.pom (3 KB at 3.9 kb/sec) downloading:https://repo.maven.apache.org/maven2/org/apache/maven/ Surefire/maven-surefire-common/2.10/maven-surefire-common-2.10.pom downloaded:https://repo.maven.apache.org/
Maven2/org/apache/maven/surefire/maven-surefire-common/2.10/maven-surefire-common-2.10.pom (4 KB at 6.8 KB/sec) downloading:https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/2.10/ Surefire-booter-2.10.jar downloading:httpS://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.10/surefire-api-2.10.jar Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.10/ Maven-surefire-common-2.10.jar downloading:https://repo.maven.apache.org/maven2/org/codehaus/plexus/ Plexus-utils/2.1/plexus-utils-2.1.jar Downloaded:https://repo.maven.apache.org/maven2/org/apache/maven/surefire /surefire-booter/2.10/surefire-booter-2.10.jar (MB at 17.1 kb/sec) downloaded:https://repo.maven.apache.org/ Maven2/org/apache/maven/surefire/surefire-api/2.10/surefire-api-2.10.jar (158 KB at 64.6 KB/sec) downloaded:https:// repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.10/ Maven-surefire-common-2.10.jar (KB at 21.7 kb/sec) downloaded:https://repo.maven.apache.org/maven2/org/codehaus/
Plexus/plexus-utils/2.1/plexus-utils-2.1.jar (KB at 16.2 kb/sec) [INFO] Tests are skipped. 
[INFO] [INFO]---maven-jar-plugin:2.4:jar (default-jar) @ StoRm-yarn---[info]------------------------------------------------------------------------[INFO] Build SUCCESS [ INFO]------------------------------------------------------------------------[INFO] Total time:24:18 min [info] Finished at:2015-03-08t22:13:01+08:00 [INFO] Final memory:13m/32m [INFO]------------------------------------------- -----------------------------[jifeng@feng02 storm-yarn-master]$ ls bin cla.pdf create-tarball.sh lib LICENSE.txt Pom . XML readme.md src target [jifeng@feng02 storm-yarn-master]$ CD target/[jifeng@feng02 target]$ ls classes g Enerated-test-sources maven-status test-classes generated-sources maven-archiver storm-yarn-1.0-al Pha.jar [jifeng@feng02 target]$ cd ...

4:storm-0.9.0 unzip and move to Storm-yarn-master sibling

[jifeng@feng02 storm-yarn-master]$ cd lib
[jifeng@feng02 lib]$ unzip  storm-0.9.0-wip21.zip        
:  Storm-0.9.0-wip21.zip
[Jifeng@feng02 lib]$ MV Storm-0.9.0-wip21. /.. /
[jifeng@feng02 lib]$ cd ...
[Jifeng@feng02 storm-yarn-master]$ cd ...
[jifeng@feng02 storm]$ ls
storm-0.9.0-wip21  storm-yarn-master  storm-yarn.zip

5: Configuration

[Jifeng@feng02 storm-yarn-master]$ VI ~/.bash_profile

Add the following:

Export storm_work= $HOME/hadoop/storm
Export Storm_home= $STORM _work
Export path= $PATH: $STORM _work/storm-yarn-master/bin: $STORM _work/storm-0.9.0-wip21/bin

[Jifeng@feng02 lib]$ Source ~/.bash_profile

Effect

6: Upload storm.zip to HDFs

[Jifeng@feng02 lib]$ Hadoop fs-mkdir-p/lib/storm/0.9.0-wip21
[jifeng@feng02 lib]$ Hadoop fs-movefromlocal storm.z Ip/lib/storm/0.9.0-wip21
[jifeng@feng02 lib]$ hadoop fs-mkdir-p/user/storm
[jifeng@feng02 lib]$ Hadoop fs-ch Own Storm/user/storm
[jifeng@feng02 lib]$ CD.
[Jifeng@feng02 storm-yarn-master]$ CD ...

7: Start
Storm-yarn Launch/home/jifeng/hadoop/storm/storm-0.9.0-wip21/conf/storm.yaml

[Jifeng@feng02 storm-0.9.0-wip21]$ CD ... [Jifeng@feng02 storm]$ Storm-yarn Launch/home/jifeng/hadoop/storm/storm-0.9.0-wip21/conf/storm.yaml slf4j:class
Path contains multiple slf4j bindings. Slf4j:found Binding in [Jar:file:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar !/org/slf4j/impl/staticloggerbinder.class] Slf4j:found binding in [jar:file:/home/jifeng/hadoop/storm/ Storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/staticloggerbinder.class] Slf4j:see http://
Www.slf4j.org/codes.html#multiple_bindings for a explanation. Slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 15/03/14 17:35:23 INFO client. Rmproxy:connecting to ResourceManager at feng01/10.6.3.84:8032 15/03/14 17:35:23 WARN. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable 15 /03/14 17:35:25 INFO yarn. Stormonyarn:copy App Master jar from local filesystem and add to local environment 15/03/14 17:35:27 INFO yarn. Stormonyarn:set the environment for the application master 15/03/14 17:35:27 INFO yarn. Stormonyarn:yarn CLASSPATH COMMAND = [[YARN, CLASSPATH]] 15/03/14 17:35:27 INFO YARN. Stormonyarn:yarn CLASSPATH = [/home/jifeng/hadoop/hadoop-2.4.1/etc/hadoop:/home/jifeng/hadoop/hadoop-2.4.1/etc/ hadoop:/home/jifeng/hadoop/hadoop-2.4.1/etc/hadoop:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/lib/* :/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/hdfs:/ home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/hdfs/lib/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/hdfs/*:/ home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/yarn/*:/ home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/ mapreduce/*:/home/jifeng/hadoop/hadoop-2.4.1/contrib/capacity-scheduler/*.jar:/home/jifeng/hadoop/hadoop-2.4.1 /share/hadoop/yarn/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/*] 15/03/14 17:35:27 INFO yarn. stormonyarn:using java_home = [/home/jifeng/jdk1.7.0_45] 15/03/14 17:35:27 INFO yarn. Stormonyarn:setting up App Master Command:[/home/jifeng/jdk1.7.0_45/bin/java,-dstorm.home=./storm/ storm-0.9.0-wip21/,-dlogfile.name=<log_dir>/master.log, Com.yahoo.storm.yarn.MasterServer, 1><log_ Dir>/stdout, 2><log_dir>/stderr] 15/03/14 17:35:28 INFO impl. yarnclientimpl:submitted application application_1425798050526_0001 application_1425798050526_0001 [jifeng@feng02  storm]$

8: Close

[Jifeng@feng02 storm]$ storm-yarn shutdown-appid application_1425798050526_0001-output ~/hadoop/storm/storm.yaml
Slf4j:class path contains multiple slf4j bindings. Slf4j:found Binding in [Jar:file:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar !/org/slf4j/impl/staticloggerbinder.class] Slf4j:found binding in [jar:file:/home/jifeng/hadoop/storm/ Storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/staticloggerbinder.class] Slf4j:see http://
Www.slf4j.org/codes.html#multiple_bindings for a explanation. Slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 15/03/15 21:13:27 INFO client. Rmproxy:connecting to ResourceManager at feng01/10.6.3.84:8032 15/03/15 21:13:28 WARN. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable 15 /03/15 21:13:28 INFO yarn. Stormonyarn:application for application_1425798050526_0001:feng03:9000 15/03/15 21:13:28 INFO yarN.stormonyarn:attaching to feng03:9000 to talk to app master application_1425798050526_0001 15/03/15 21:13:29 INFO Yarn.S TormMasterCommand:org.apache.thrift7.transport.TTransportException