Single-host installation of Storm in CentOS 6.5

Single-host installation of Storm in CentOS 6.5

This article describes how to install and configure Twitter Storm as your own notes.
Storm installation instructions (e): https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
Storm is installed in standalone and cluster versions, but the configuration is slightly different, roughly the same.
To use storm, install the following tools:
Python, zookeeper, zeromq, jzmq, and storm

Step 1: Install Python2.7.2
Wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz
Tar zxvf Python-2.7.2.tgz
Cd Python-2.7.2
./Configure--without-libsodium
Make
Make install
Vi/etc/ld. so. conf
Append/usr/local/lib/
Sudo ldconfig

Step 2: Install zookeeper

Wget http://labs.mop.com/apache-mirror/zookeeper/zookeeper-3.3.5/zookeeper-3.3.5.tar.gz
Tar-zxvf zookeeper-3.3.5.tar.gz
Cp-R zookeeper-3.3.5/usr/local/
Ln-s/usr/local/zookeeper-3.3.5 // usr/local/zookeeper
Vim/etc/profile (set ZOOKEEPER_HOME and ZOOKEEPER_HOME/bin)
Export ZOOKEEPER_HOME = "/path/to/zookeeper"
Export PATH = PATH: ZOOKEEPER_HOME/bin
Cp/usr/local/zookeeper/conf/zoo_sample.cfg/usr/local/zookeeper/conf/zoo. cfg (use zoo_sample.cfg to create $ ZOOKEEPER_HOME/conf/zoo. cfg)
Mkdir/tmp/zookeeper
Mkdir/var/log/zookeeper
The single-host installation of zookeeper is complete.

Step 3: Install zeromq and jzmq
Jzmq installation may depend on zeromq. Therefore, you should first install zeromq and then jzmq.
1) install zeromq:
Wget http://download.zeromq.org/zeromq-2.2.0.tar.gz
Tar zxf zeromq-2.2.0.tar.gz
Cd zeromq-2.2.0
./Configure
Make
Make install
Sudo ldconfig (update LD_LIBRARY_PATH)
Zeromq installation is complete.
NOTE: If any dependency error is reported, you need to install it:
Jzmq dependencies dependency package
Sudo yum install uuid *
Sudo yum install libtool
Sudo yum install libuuid
Sudo yum install libuuid-devel
2) install jzmq
Yum install git
Git clone git: // github.com/nathanmarz/jzmq.git
Cd jzmq
./Autogen. sh
./Configure
Make
Make install
Then jzmq is installed.
Note: In. /autogen. sh: autogen. sh: error: cocould not find libtool is required to run autogen. sh, because libtool is missing, you can use # yum install libtool.

Step 4: install Storm
Wget http://cloud.github.com/downloads/nathanmarz/storm/storm-0.8.1.zip
Unzip storm-0.8.1.zip
Mv storm-0.8.1/usr/local/
Ln-s/usr/local/storm-0.8.1 // usr/local/storm
Vim/etc/profile
Export STORM_HOME =/usr/local/storm-0.8.1
Export PATH = PATH: STORM_HOME/bin
So far, the single-host version of Storm has been installed.

Step 5: Test WordCount in local mode.
Download storm-starter compilation and import the eclipse project:
()
1. Download the strom starter code git clone https://github.com/nathanmarz/storm-starter.git
2. Use mvn-f m2-pom.xml package for compilation
3. Copy m2_pom.xml in the storm-starter directory to pom. xml, because eclipse requires pom. xml
4. Use mvn eclipse: eclipse to compile it into an eclipse project
5. In Eclipse, select the storm-starter path for import. Generally, after importing a project, you need to set the appropriate M2 _ to check whether the project is correct. You may need to configure the M2_REPO variable,
M2_REPO configuration method: Right-click the project and choose Properties> Java Build Path> Libraries> AddVariable> Configure Variable> New
Enter Name: M2_REPO, Path: localRepository Path-> OK to refresh the project. The code is correct and can be developed.
6. After the compilation is correct, run WordCountTopology in the storm. starter directory locally. The screenshot below shows that the local mode can run through
Use the export function of eclipse to export the jar package of the project, so that the corresponding logic can be submitted in Distributed scenarios.
Strom-Starter fails to build and the twitter4j package is missing:

Modify the pom file m2-pom.xml for Storm-Starter and modify the dependency versions of the twitter4j-core and twitter4j-stream packages in dependency, as shown below:

Org. twitter4j
Twitter4j-core
[2.2 ,)

Org. twitter4j
Twitter4j-stream
[2.2 ,)
Note that the above running WordCount in local mode does not actually use the tool installed above. It is just a testing demo in a storm virtual environment. So how can we run the program in the just-built standalone environment,
Simple. Official example:
Note: In the official instance, if the WordCountTopology class does not contain the parameter, it is actually the local mode of execution, that is, the virtual environment just mentioned. With the parameter included, the jar is sent to storm for execution.
First, set the environment:
Start zookeeper:
The/usr/local/zookeeper/bin/zkServer. sh standalone version is directly started without modifying any configuration. For example, you need to modify zoo. cfg in the cluster.
Configure storm:
File in/usr/local/storm/conf/storm. yaml
Content:
Storm. zookeeper. servers:
-127.0.0.1
Storm. zookeeper. port: 2181
Nimbus. host: "127.0.0.1"
Storm. local. dir: "/tmp/storm"
Supervisor. slots. ports:
-6700
-6701
-6702
-6703
This script file is not written in any way, so you must add a space at the beginning of each item during configuration, and then add a space after the colon, otherwise, storm will not recognize this configuration file.
Note: storm. local. dir indicates the local directory used by storm. Nimbus. host indicates that the machine is a master machine, that is, nimbus. Storm. zookeeper. servers indicates which machines are zookeeper servers. Storm. zookeeper. port indicates the zookeeper port number, which must be consistent with the port number configured by zookeeper. Otherwise, a communication error may occur. Remember to remember. You can also configure a superevisor. slot. port, supervisor. slots. ports indicates the number of worker slots of the supervisor node, that is, a maximum of several worker processes can be run (each sprout or bolt starts only one worker by default, but can be changed to multiple by using conf ).
Run:

# Bin/storm nimbus (start the master node)
# Bin/storm supervisor (start slave node)
Run the following command:

# Storm jar StormStarter. jar storm. starter. WordCountTopology test
The function of this command is to use storm to send the jar to storm for execution. The test is the defined toplogy name.
After this is done, the task is sent to storm for running. You can also run the following command:

# Bin/storm ui
Start the ui. You can use the browser, ip: 8080/to view the running I.

Thoughts on a scalable real-time data processing architecture based on Storm

How does Storm allocate tasks and load balancing?

Storm Process Communication Mechanism Analysis

Apache Storm History and Lessons

For details about Apache Storm, click here
Apache Storm: click here