Storm document----Build your own storm cluster

Reprint Please specify source: http://blog.csdn.net/beitiandijun/article/details/41802543

Ubuntu under Storm installation steps

Before installing storm, you first need to install some dependent libraries:

Zookeeper, JDK 6, python2.6.6, JZMQ, ZEROMQ

The dependent libraries required for these libraries are no longer one by one-sum.

The following are the specific installation procedures:

First, install the JDK

Zookeeper requires JDK 6 or later (currently the latest stable version is JDK8), but because Storm requires JDK 6 to be installed, you choose to install JDK 6.

Jdk:http://www.oracle.com/technetwork/java/javase/archive-139210.html

JDK installation process: Http://www.oracle.com/technetwork/java/javase/index-137561.html#linux

1. Download JDK6

(1) Open JDK:

(2) Select Java SE 6 and download

2. Installing the JDK

(1) Modify permissions:

chmod+x 6u <version> -linux-x64.bin

(2) Installation: "Java Core Programming", published by Sun, is recommended for installation in/usr, not/usr/local, for specific reasons.

sudo./ 6u <version>-linux-x64.bin installation directory

(3) Configuring environment variables

Export Java_home= installation directory

Export Jre_home= $JAVA _home/jre

Export classpath=.: $JAVA _home/lib: $JRE _home/lib

Export path= $JAVA _home/bin: $PATH

Note: Solve JDK version conflicts are modified by modifying these environment variables

Three different ways:

Mode one: Available only in the current shell:

The above statement in the current shell command line

Mode two: Available only in the current user:

Add the above statement at the end of the ~/.BASHRC

Enabled in the current shell:

SOURCE ~/.BASHRC

Method Three: All users are available:

Add the above statement at the end of the/etc/profile

Enabled in the current shell:

sudo source/ect/profile

In this example, mode three is used.

3. Test if Java is installed successfully

Java-version

Second, installation zookeeper

Reference Zookeeper official wiki:http://zookeeper.apache.org/doc/r3.3.3/zookeeperadmin.html

1, zookeeper system requirements:

Supported platforms include: Gnu/linux, Sun Solaris, FreeBSD, Win32, MAXOSX

Software Requirements: JDK 6 or higher is required, for zookeeper clusters, the official recommended minimum number of nodes is 3, and run on different machines; to avoid performance degradation due to data swap, increase the Java heap size as much as possible in memory, such as 4GB Memory Java heap size can be set to 3GB

2, the specific installation process

(1)

(2) Installing Zookeeper's Dependent Toolkit:

GCC, Cppunit, python-setuptools

I. GCC system usually comes with, do not need to install

II. Installation Python-setuptools:

Sudoapt-get Install Python-setuptools

III. Installing the Cppunit

Sudoapt-get Install Cppunit

(2) Configure Zookeeperconfiguration file:

Since there are 10 nodes in this cluster, the configuration

Third, install Python

Depending on storm requirements, installation of python2.6.6,python:http://www.python.org/ftp/python/2.6.6/python-2.6.6.tar.bz2 is required

1. Download Python 2.6.6

wget http://www.python.org/ftp/python/2.6.6/Python-2.6.6.tar.bz2

2. Compiling and installing Python 2.6.6

(1) Decompression: tar-jxvfpython-2.6.6.tar.bz2

(2) Compile and install:

CD Python

Sudo./configure--prefix=/usr/local/lib/python2.6.6

Sudomake

Several errors may occur during the installation process

Error One:

Based on the above error, the search results are due to the multiarch mechanism of Debian/ubuntu: even binaries compiled under different schemas can be run under the same system.

Reference Documentation:

Http://jisedai.vayn.de/2011/06/28/python-3-installation-note.html

http://bugs.python.org/issue9762

http://bugs.python.org/issue11715

Cause of error: The search paths for libraries under different schemas may not be consistent, so setup.py may not find the library files that are needed at compile time.

Workaround: Modify the setup.py to add the Multiarch path to the setup.py search path.

Specific Patch: https://hg.python.org/cpython/rev/bd0f73a9538e

Error Two:

After the error is resolved, make, still error:

The above error indicates that a dependent library is missing: Follow the prompts to indicate that the dependent libraries above can modify the Detect_modules () function in setup.py to cancel the dependency, but because it is not possible to anticipate which results are not installed, it is decided to install a total of 18 dependent libraries.

Install in sequence:

1) Install zlib (compression library)

Find Zlib Library

Installation: sudo apt-get install Zlib1g-dev

To see if the installation was successful, Zlib did not appear stating that the installation was successful, continue the installation

2) Install bz2

Find bz2 Library (compression library)

Installation: Sudoapt-get Install Libbz2-dev

3) Install GDBM (dbm is a database of Unix-like systems, use hash to save unstructured data, does not support SQL,GDBM is the GNU version of DBM)

Find gdbm and install:

Sudoapt-get Install Libgdbm-dev

Note: After installing GDBM, you do not need to install dbm again

4) Install ReadLine (GNU, for edit command line, Current User: Gdb,vi,emacs, etc.)

Find and install ReadLine:

Sudoapt-get Install Libreadline-dev

5) Install _sqlite3 (SQL Database engine C Library )

Installation: Libsqlite3-dev

6) Install _curses and _curses_panel

Installation: Libncurses5-dev

3, installation failure of the dependent library description

The following is a dependent library that did not install successfully: (does not mean that the installation failed, but that python2.6.6 cannot call the following libraries)

First, the installation steps are listed as follows:

1) Install _BSDDB (Pythoninterface for Berkeley DB)

Method One: unsuccessful

Search Apt-cachesearch BSDDB

But after installing PYTHON-BSDDB3 and python-bsddb3-dbg, execute Python's make discovery still hints that there is no _bsddb, so go to the Python website for BSDDB related information:

Https://pypi.python.org/pypi?%3Aaction=search&term=bsddb&submit=search

Method Two: unsuccessful

Still not, online search found that someone first installed Berkeley Db:libdb-dev

Method Three:

Online has said Bsddb is a very old Berkeley DB module that can not be installed.

By modifying the setup.py

disabled_module_list= ["bsddb185"]

Still fail

2) Install _hashlib (hash table) and _ssl (information security, data encryption)

Search and install: Sudoapt-get Install Libssl-dev

After installing Libssl-dev, the compiled Python still does not successfully identify the _ssl module.

Installing Openssl:sudo apt-get Install OpenSSL

The result still fails to successfully invoke the _ssl module, as shown in the following:

3 ) DL , Imageop , Linuxaudiodev , Ossaudiodev

DL is an obsolete module (replaced by cTYPES) on a 32-bit system;

Imageop is also an obsolete module on 32-bit systems, replaced by PIL;

Linuxaudiodev is an audio module;

Ossaudiodev is an audio module;

_tkinter is the interface module.

These modules do not need to be installed.

4. Resolving Python version conflicts

Originally there is python2.7 on the machine, now storm needs to use python2.6, how to solve the version conflict problem is need to solve.

Search the Internet and propose two solutions:

(1) Install Python version management tool Pythonbrew

: https://pypi.python.org/pypi/pythonbrew/1.3.5

Download Unzip, enter pythonbrew1.3.5

To read the Readme.rst, follow the recommended installation steps:

(1) Curl-klhttp://xrl.us/pythonbrewinstall | Bash

Need to install dependent libraries: curl

Sudoapt-get Install Curl

After you install curl, follow the steps (1) command line to display:

(2) According to the above output hint, the prompt command is written to the end of ~/.BASHRC, namely:

[-S "$HOME/.PYTHONBREW/ETC/BASHRC"]] && source "$HOME/.PYTHONBREW/ETC/BASHRC"

(3) Select the appropriate version number:

Pythonbrew Switch 2.6.6

Note: Pythonbrew software is the key recommendation , I hope you install pythonbrewbefore installing python , you can use pythonbrew Install the desired python version.

(2) using soft Links:

In this example, python2.6.6 is installed under/usr/local/lib, so a soft link can be:

sudo ln-s/usr/bin/python/usr/local/lib/python2.6.6/bin/python2.6

Iv. installation of ZEROMQ

1, installation Zeromq

(1) Download ZEROMQ source code, version number 2.1.7

wget http://download.zeromq.org/zeromq-2.1.7.tar.gz

Error, Missing Uuid-dev

So install the sudo apt-get install Uuid-dev

(2) Compile and install:

Make

Make install

V. Installation of JZMQ

(1) Download JZMQ source code

Https://github.com/zeromq/jzmq

Execute./autogen.sh, Error:

Need to install Pkg-config

sudo apt-get install Pkg-config

Still error:

Need to install Libtool

sudo apt-get install Libtool

Still error:

Need to install autoconf and Automake:

sudo apt-get install Automake autoconf

The dependent libraries are automatically installed during the installation M4

./autogen.sh execution succeeded, generating configure

./configure

Make error:

Workaround: http://my.oschina.net/mingdongcheng/blog/43009

1) A classdist_noinst.stamp empty file is generated first in the makefile path and the SRC directory.

2) to Src/org/zeromq directory: Javac *.java

3) to src directory: make-j24; Make install

4) to makefile under the path: make-j24; make install

Vi. storm Installation and configuration process:

1. Download the executable package:

Http://storm.incubator.apache.org/downloads.html

2. Startup process:

Refer to Storm official website:

Http://storm.incubator.apache.org/documentation/Setting-up-a-Storm-cluster.html

(1) Modify the Storm configuration file:

(2) Start Storm Nimbus:

Bin/storm Nimbus >/dev/null 2>&1 &

(3) Start Supervisor:

Bin/storm Supervisor >/dev/null 2>&1 &

(4) Start the UI:

Bin/storm UI >/dev/null 2>&1&

Attention:

UI node and Nimbus nodes are best located on the same node .

UI The default use port is 8080 , but this port has been TCP needs to be modified. Ui.port parameter to prevent port collisions.

(5) To view the Storm UI:

Storm document----Build your own storm cluster