Centos6.5 install storm Cluster

1First install the dependency package

[[email protected] ~]# yum install uuid*[[email protected] ~]# yum install libuuid[[email protected] ~]# yum install libuuid-devel[[email protected] ~]# yum install libtool[[email protected] ~]# yum install e2fsprogs[[email protected] ~]# yum install e2fsprogs-devel

 

2Install zeromq

Download: http://download.zeromq.org/zeromq-4.1.2.tar.gz

Unzip: Tar xzf zeromq-4.1.2.tar.gz

Compile and install:

./configure --without-libsodiummakemake installsudo ldconfig

 

3Install jzmq (first install JDK)

Install git:

yum install gitgit clone git://github.com/nathanmarz/jzmq.gitcd jzmq

Or

If you do not install git, use wget:

wget https://codeload.github.com/zeromq/jzmq/zip/masterunzip  jzmq-master.zipcd jzmq-jni/

Compile and install:

./autogen.sh./configuremakemake install

 

4Install Python (skip this step if centos comes with python2.6)

wget https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tgztar -zxvf Python-3.5.0.tgzcd Python-2.7.2./configuremakemake install

After installation:

VI/etc/lD. So. conf append/usr/local/lib/

 

5Install zookeeper

Reference: http://www.cnblogs.com/hunttown/p/5452138.html

 

6Install storm

Download: http://mirror.bit.edu.cn/apache/storm/apache-storm-0.9.6/apache-storm-0.9.6.tar.gz

Unzip: tar-zxvf apache-storm-0.9.6.tar.gz

Set environment variables:

[[Email protected] ~] # Vim/etc/profile # stormport storm_home =/home/hadoopuser/apache-storm-0.9.6export Path = $ path: $ storm_home/bin [root @ Hadoop-NN-01 ~] # Source/etc/profile # make the environment variable take effect

 

7, Configuration

Storm configuration document: https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml

[[Email protected] conf] # Vim storm. yaml # Pay Attention to the yaml format requirements. Note that a space must be added at the beginning of each item, and a space must be added after the colon. Otherwise, storm will not know the configuration file storm. zookeeper. servers:-"Zookeeper-01"-"Zookeeper-02"-"Zookeeper-03" # Storm. zookeeper. port: 2181 # configure Nimbus if the port is not the default port. HOST: "Hadoop-NN-01" Nimbus. thrift. port: 6627storm. zookeeper. port: 2181storm. local. dir: "/home/hadoopuser/Apache-storm-0.9.6/tmp" supervisor. slots. ports:-6700-6701-6702 # To Ni Mbus 1 GB is enough for Nimbus. childopts: "-xmx1024m" # To supervisor 1 GB is enough for supervisor. childopts: "-xmx1024m" # To worker jobs can generally be larger because they need to process data. Generally, 4 GB or above is used. Worker. childopts: "-xmx4096m"

 

Storm clusters have two types of nodes: one is a control node (Nimbus node) and the other is a worker node (supervisor node ). All topology tasks must be configured on the storm client node ~ The/. Storm/storm. yaml file is assigned by the nimbus node to other supervisor nodes for processing.

mkdir ~/.stormcp /home/hadoopuser/apache-storm-0.9.6/conf/storm.yaml ~/.storm/storm.yaml

 

# Modify hosts (Omitted ):

Changed the nimbus node hosts to nimbushost.

Change the supervisor node hosts to supervisorhost1 ...... X

 

# Process description:

Zkserver Nimbus, supervisor and UI

Storm Nimbus & wait for a while until ---- backtype. Storm. daemon. Nimbus appears

Storm Supervisor & wait for a while until ---- storm. daemon. Supervisor appears

Storm UI & wait for a while until ---- properties backtype. Storm. UI. Core appears

JPS allows you to view the running status of each process.

The process corresponding to core is storm UI.

The process corresponding to JPS is Java JPs.

The process corresponding to Nimbus is storm Nimbus.

The supervisor processes are storm supervisor.

The process corresponding to quorumpeermain is zkserver. Sh.

 

8Start storm

# Start zookeeper: [[email protected] ~] # Zkserver. Sh start # Start STORM: [Ro [email protected] ~] # Storm Nimbus & # Start in the background [[email protected] ~] # Storm Supervisor & [[email protected] ~] # Storm UI &

 

Check the status in the browser: http: // 192.168.107.82: 8080/

 

9, Submit a job (Topology)

[[email protected] ~]# storm jar xxxx.jar storm.starter.WordCountTopology arg1 arg2 arg3

 

10Disable a job

[[email protected] ~]# storm kill topologyname

 

Note:

1. After the storm background process is started, log files of each process will be generated under the logs/subdirectory under the storm installation and deployment directory.

2. After testing, the storm UI and storm Nimbus must be deployed on the same machine; otherwise, the UI will not work properly because the UI process will check whether the native has a nimbus link.

3. For ease of use, bin/storm can be added to system environment variables.

 

Centos6.5 install storm Cluster