Storm Cluster setup and testing

Install storm

Steps:

A, Build zookeeper cluster

B. Install storm-dependent software

C. Installing and configuring Storm

1, Build zookeeper cluster

2. Install storm-dependent software

ZeroMQ 2.1.7 is a message processing queue library that elastically scales between multiple threads, cores, and host boxes.

JZMQ underlying Message Queuing

Java 6

Python 2.6.6

Unzip//For decompression can not be installed

ZEROMQ Installation:

wget http://download.zeromq.org/zeromq-2.1.7.tar.gz

TAR-XZVF zeromq-2.1.7.tar.gz

CD zeromq-2.1.7

./configure

Make

sudo make install

JZMQ Installation:

git clone https://github.com/nathanmarz/jzmq.git

CD JZMQ

./autogen.sh

./configure

Make

sudo make install

Possible error and resolution during installation:

1. Error occurred: Configure:error:Unable to find a working C + + compiler

You need to install the g++ compiler:

sudo yum install gcc-c++

2, error occurred: Configure:error:cannot link with-luuid, install Uuid-dev.

Need to install packages Uuid-devel and Libuuid-devel:

sudo yum install Uuid-devel

sudo yum install Libuuid-devel

3. An error occurred: Autogen.sh:error:could not find Libtool. Libtool is required to run autogen.sh.

Need to install Libtool:

sudo yum install Libtool

Possible commands to use:

Yum Install-y gcc-c++.x86_64

Yum Install-y uuid-devel.x86_64

Yum Install-y libuuid-devel.x86_64

Yum Install-y libtool.x86_64

Yum install-y git.x86_64//JZMQ installation using the tools git

Install storm

Download Unzip:

wget https://dl.dropboxusercontent.com/s/tqdpoif32gufapo/storm-0.9.0.1.tar.gz

Tar xzvf. /storm-0.9.0.1.tar.gz

Conf/storm.yaml in the configuration file, start with a space:

Storm.zookeeper.servers:

-"Master"

storm.zookeeper.port:2181 "Default"

Nimbus.host: "Master"

Storm.local.dir: "/opt/soft/storm-0.9.0.1/data"

Supervisor.slots.ports:

-6700

-6701

-6702

-6703

Java.library.path:/usr/local/lib:/opt/local/lib:/usr/lib "Default"

ui.port:8081

Parameter interpretation

Storm.zookeeper.servers: This is a list of hosts for the zookeeper cluster configured for the Storm cluster

Storm.local.dir:Nimbus and Supervisor daemons require a local disk directory to store a small amount of state (like Jars,confs, others), each machine creates these directories and assigns writable permissions

Java.library.path: This is the local library (ZEROMQ and JZMQ) load path used by storm. Most installations, the default path is "/usr/local/lib:/opt/local/lib:/usr/lib" on the line, so you may not need to configure it.

Supervisor.slots.ports: For each worker machine, you use this configuration to specify how many workers run on that machine. Each worker receives a message using a single port, and this setting defines which port is open and can be used. If you define 5 ports, then storm will allocate 5 workers on this machine.

Storm Start test

Storm is a fast-failing (fail-fast) system, which means that these processes can stop at any time due to an error. Because Storm is designed, it is safe to stop at any time, and when the process restarts correctly, it recovers. This is why storm keeps the process stateless--if Nimbus or supervisors restarts, the running topologies is unaffected.

Nohup hanging to the background execution

1) Nimbus

Run the command "Bin/storm Nimbus" under the supervision of the Master machine

2) Supervisor

Run the command "Bin/storm supervisor" under the supervision of each worker machine. The Supervisor Daemon is responsible for starting and stopping worker processes on that machine.

3) UI

Run the supervision command "Bin/storm UI" to run the Storm UI (you can access a site from a browser that provides diagnostic information for clusters and topologies). Enter "Http://{nimbus host}:8081" in your browser to access the UI.

Add

Master Nodehang to background bootnohup./storm Nimbus & Start UInohup./storm UI & the UI can be started by a browser to access thehttp://master:8081   You can start a supervisor on the Nimbus.  slavel Start nohup./storm Supervisor &

Storm Cluster setup and testing