Build Jstrom on virtual machines

Original: http://blog.csdn.net/u014134180/article/details/51810311

• One installation step
• Two building zookeeper cluster
  • 1 ZooKeeper stand-alone installation and configuration
    • 11 Download Zookeeper installation package
    • 12 Unpacking the Zookeeper installation package
    • 13 Setting environment variables
    • 14 Configuring Zoocfg
    • 15 Start Zookeeper
• Three building Jstorm Cluster
  • 1 Installing Python 26
  • 2 Installing JDK version 17
  • 3 Installing Jstorm
    • 31 Configuring Jstorm_homeconfstormyaml
    • 32 examples
    • 33 execution on the node committing the jar
  • 4 Installing Jstorm Web UI4 on a tomcat server
  • 5 Start Jstorm
• Four more links
  • 1 Zookeeper Knowledge More links
  • 2 Jstorm Knowledge More links

One installation step

• Download Relase package (jstorm-2.1.1.zip) from Downloads
• To prepare the Java JDK first
• Build Zookeeper Cluster
• Installing Python 2.6
• Configure $jstorm_home/conf/storm.yaml
• Building a Web UI
• Start Jstorm Cluster

Two building zookeeper cluster

• Installation Steps trouble reference "Zookeeper easy installation Step 1"
• Zookeeper Configuration Trouble Reference "zookeeper stand-alone mode, pseudo-cluster mode, cluster mode installation 2"

2.1 ZooKeeper stand-alone installation and configuration 2.1.1 Download ZooKeeper installation package

: http://apache.dataguru.cn/zookeeper/zookeeper-3.4.3/zookeeper-3.4.3.tar.gz
What I download here is: zookeeper-3.4.5.tar.gz

2.1.2 Unpacking the Zookeeper installation package

wu_being@JStorm-VirtualBox:~$ sudo tar -zxvf zookeeper-3.4.5.tar.gzwu_being@JStorm-VirtualBox:~$ sudo chown -R wu_being:wu_being zookeeper-3.4.5

• 1
• 2

• 1
• 2

2.1.3 Setting Environment variables

Add the following information after The/etc/profile AND/HOME/WU_BEING/.BASHRC files 3:

JAVA_HOME=/home/wu_being/jdk1.7.0ZOOKEEPER_HOME=/home/wu_being/zookeeper-3.4.5CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$ZOOKEEPER_HOME/libPATH=$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$PATHexport ZOOKEEPER_HOMEexport JAVA_HOMEexport CLASSPATHexport PATH

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9

after setting the environment VARIABLE/HOME/WU_BEING/.BASHRC file, you want to update the source:
[email protected]:~$source /home/wu_being/.bashrc
[email protected]:~$source /etc/profile

2.1.4 Configuration Zoo.cfg

The configuration file is stored in the $zookeeper_home/conf/directory, the Zoo_sample.cfd file name is changed to Zoo.cfg ( cp zoo_sample.cfg zoo.cfg ), and the default configuration is as Follows:

# of milliseconds of each tickticktime=2000# The number of ticks that initial# Synchronization phase can takeinitlimit=10# The number of ticks that can pass between# Sending a request and getting an acknowledgementsynclimit=5# The directory where the snapshot is Stored.# do not use/tmp for storage,/tmp here are just# example Sakes. # # # # #这是我修改后的dataDir =/home/wu_being/zookeeper-3.4.5/data # the port at which The clients would connectclientport=2181## Be sure to read the maintenance sections of The# Administrator Guide b Efore turning on Autopurge. ## http://zookeeper.apache.org/doc/current/ Zookeeperadmin.html#sc_maintenance## the number of snapshots to retain in Datadir #autopurge. snapretaincount=3 #autopurge. purgeinterval=1        

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21st
• 22
• 23
• 24
• 25

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21st
• 22
• 23
• 24
• 25

Configuration Description:
ticktime: This time is the time interval between the Zookeeper server or between the client and the server to maintain the heartbeat, that is, each ticktime time sends a heartbeat.
DataDir: As the name implies is Zookeeper to save the data directory, by default, Zookeeper will write the data log file is also stored in this directory.
clientport: This port is the port that the client connects to the Zookeeper server, Zookeeper listens to the port and accepts the Client's access Request.

2.1.5 Start Zookeeper

When these configuration items are configured, You can now start zookeeper:

Wu_being@JStorm-VirtualBox: ~/netstat-at|grep2181#查看zookeeper端口wu_being@JStorm-VirtualBox: ~/netstat-nat #查看端口信息wu_being  @JStorm- Virtualbox:~/zookeeper-3.4. 5/bin$./zkserver.sh start  #启动 wu_being @JStorm-virtualbox:~/zookeeper-3.4. 5/bin$ JPS  #查看启动的服务名称wu_being Span class= "hljs-variable" > @JStorm-virtualbox:~/ Zookeeper-3.4. 5/bin$./zkserver.sh stop  #关闭      

• 1
• 2
• 3
• 4
• 5

• 1
• 2
• 3
• 4
• 5

Note: JPS command is Java jdk, start Zkserver is also to prepare the Java jdk, or start success is false phenomenon!!!

• Jps itself service name: 4793 Jps
• Zkserver Service name: 4464 Quorumpeermain

After starting to check whether the Zookeeper is already in service, netstat -at|grep 2181 You can check through the command to see if there is a ClientPort port number in the listener Service. And in zookeeper-3.4.5 generate Data folder, Data folder Version-2 folder and Zookeeper_server.pid File.

Three-building jstorm Cluster 3.1 installation python 2.6

• If Python is available on the current system, you may not have to install Python
• You can refer to Python
• You can also use Https://github.com/utahta/pythonbrew to install python > curl-kl http://xrl.us/pythonbrewinstall | Bash

-s $HOME/.pythonbrew/etc/bashrc && source $HOME/.pythonbrew/etc/bashrcpythonbrew install 2.6.7pythonbrew switch 2.6.7

• 1
• 2
• 3

• 1
• 2
• 3

3.2 Installing the JDK (version 1.7)

Note that if the current system is a 64-bit system, you will need to download the 64-bit jdk, and if 32 is the system, download the 32-bit JDK

3.3 Installing Jstorm

Assuming Jstorm-0.9.6.3.zip as an example

unzip jstorm-2.1.1.zipvi ~/.bashrcexport JSTORM_HOME=/home/wu_being/jstorm-2.1.1export PATH=$PATH:$JSTORM_HOME/bin

• 1
• 2
• 3
• 4

• 1
• 2
• 3
• 4

note: to [email protected]:~$ source .bashrc

3.3.1 Configuration $jstorm_home/conf/storm.yaml

Configuration Items:

1. Storm. Zookeeper. servers: represents the address of the zookeeper,2. Nimbus. host: represents the address of the Nimbus3. Storm. Zookeeper. root: represents the root directory of Jstorm in zookeeper, which is required when multiple jstorm share a zookeeper, which is "/jstorm" by default4. storm.local.dir: Represents the Jstorm temporary data storage directory, you need to ensure that the Jstorm program has write permissions to the directory 5. java.library< installation directory for span class= "hljs-preprocessor" >.path:zeromq and Java Zeromq library, default 6. supervisor.slots.ports: represents the list of port slots provided by supervisor, Note that there is no conflict with other ports, 68xx by default, And Storm is 67xx7 topology.classloader:false, default off classloader, If the applied jar conflicts with Jstorm's dependent jar, For example, the app uses thrift9, but when Jstorm uses thrift7, it needs to turn on Classloader. It is recommended that this option be turned off at the cluster level by default on topology that require Isolation. 

• 1
• 2
• 3
• 4
• 5
• 6
• 7

• 1
• 2
• 3
• 4
• 5
• 6
• 7

3.3.2 Example

########### these must be filled on for a storm configuration storm. Zookeeper. servers:-"172.17.21.33" Storm. Zookeeper. root:"/jstorm" Nimbus.host:  "172.17.21.33" Nimbus .host.start.childopts:  "-Xmx256m" Supervisor.childopts:  "-xmx256m" worker.childopts: "-xmx128m "storm.local< Span class= "hljs-preprocessor" >.dir:  "%jstorm_home%/data" supervisor.slots.ports:-6800-6801-6802-6803     

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15

3.3.3 execution on the node committing the jar

#mkdir ~/.jstorm#cp -f $JSTORM_HOME/conf/storm.yaml ~/.jstorm

• 1
• 2

• 1
• 2

3.4 Installing Jstorm Web UI4 on a tomcat server

You must use Tomcat 7.0 or later, and be careful not to forget to copy ~/.jstorm/storm.yaml

Web UI can not be the same node as Nimbus

mkdir ~/. jstormcp-f $JSTORM _home/conf/storm.yaml ~/.jstormtar-xzf apache-tomcat-7.0.70.tar.gz ## #cd Apache-tomcat-7.0.70cd webapps CP $JSTORM _home/jstorm-ui-2.1.1 .war./mv ROOT root.old ## #ln-s jstorm-ui-2.1.1 ROOT ## #cd. /bin./startup.sh #####     

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10

Attention:

• Download Tomcat 7.x (take apache-tomcat-7.0.70 as an Example)
• LN is not a ln-s Jstorm-ui-2.1.1.war ROOT, so be careful.
• After startup.sh, the terminal input JPS will be one more service name: 4777 Bootstrap
• Enter "local ip:8080" in the browser to manage Jstorm clusters in the Web. the prerequisite is to prepare the .jstorm/storm.yaml file and start the zookeeper, preferably starting the Nimbus and supervisor servers First.

3.5 Start Jstorm

• Perform on the Nimbus node nohup jstorm nimbus & to see if the $jstorm_home/logs/nimbus.log check is Error-free
• Perform on the Supervisor node nohup jstorm supervisor & to see if the $jstorm_home/logs/supervisor.log check is Error-free

Four more links 4.1 zookeeper knowledge more links

• What is ZooKeeper?
• Why to use Zookeeper
• Zookeeper series Textbook

4.2 Jstorm Knowledge More links

• Alibaba/jstorm
  Https://github.com/alibaba/jstorm/wiki
• Jstorm Introduction
  https://yq.aliyun.com/articles/2893
• Jstorm 2.1.1 Release
  https://yq.aliyun.com/articles/25391
• Comparison of Kafka, RabbitMQ and ROCKETMQ message middleware--message sending performance
  https://yq.aliyun.com/articles/25385
• The MapReduce of Hadoop
  http://blog.csdn.net/wangloveall/article/details/21407531

Build Jstrom on virtual machines