How to build a Storm0.8.2 environment in CentOS

Storm depends on Zookeeper and ZeroMQ. You also need to have Java and Python in your system environment. The entire setup process is as follows:

1. Build a Zookeeper cluster.
2. Install the same environment (ZeroMQ, JZMQ, Java, Python, etc.) on the control node [Nimbus] and working node [Supervisor)
3. Install the Storm framework on the control node [Nimbus] and working node [Supervisor]
4. Configure Storm through the storm. yaml file
5. Use commands to start Storm (Nimbus, Supervisor, and ui must be started separately)


1. Set up Zookeeper

Because storm does not use zookeeper to transmit messages. Therefore, the load on zookeeper is very low. The zookeeper of a single node is sufficient in most cases. Therefore, we set up a single node zookeeper.
Download the latest 3.4.5: http://zookeeper.apache.org/releases.html#download
Unzip, add the configuration file zoo. cfg under conf, you can refer to the configuration modification of the zoo-sample.cfg, mainly modify the dataDir path location

Start zookeeper:

Bin/zkServer. sh start

After starting, you can use:

Bin/zkCli. sh-server 127.0.0.1: 2181

Verify the startup status and client connection status

2. Build ZeroMq2.1.7:
We officially mentioned that version 2.1.0 should not be installed. If you encounter problems with version 2.1.7, try to downgrade to version 2.1.4.
Download: http://download.zeromq.org/
After decompression, run the following command in the directory:

Shell/>./configure


If you encounter a uuid-dev error, you need to install the uuid package:
Install uuid-dev in ubuntu and install it on centos:

Yum install libuuid-devel.x86_64

Then execute configure.
Then:

Make
Sudo make install

3. Install JZMQ
Download: https://codeload.github.com/nathanmarz/jzmq/zip/master
Decompress the package and run the following command:

./Autogen. sh
./Configure
Make
Sudo make install

./When configure is enabled, check whether your JAVA_HOME is correct. If it is incorrect, an error is returned and a prompt is displayed.

4. Install Storm

Download: https://github.com/nathanmarz/storm
Decompress and configure the storm. yaml configuration file. Main configuration:
1) storm. zookeeper. servers: Address of the Zookeeper cluster used by the Storm cluster. The format is as follows:
Storm. zookeeper. servers:
-"111.222.333.444"
-"555.666.777.888"
If the Zookeeper cluster uses no default port, the storm. zookeeper. port option is also required.
2) storm. local. dir: Nimbus and Supervisor processes are used to store a small number of states, such as local disk directories such as jars and confs. You need to create this directory in advance and grant sufficient access permissions. Configure the Directory in storm. yaml, for example:
Storm. local. dir: "/home/admin/storm/workdir"
3) java. library. path: The loading path of the local library (ZMQ and JZMQ) used by Storm. The default value is "/usr/local/lib:/opt/local/lib:/usr/lib ", generally, ZMQ and JZMQ are installed under/usr/local/lib by default, so you do not need to configure them.
4) nimbus. host: The address of the Nimbus machine in the Storm cluster. The worker nodes of each Supervisor need to know which machine is Nimbus to download files such as jars and confs of Topologies, for example:
Nimbus. host: "111.222.333.444"
5) supervisor. slots. ports: for each worker node of the Supervisor, you must configure the number of workers that the worker node can run. Each worker occupies a separate port to receive messages. This configuration option defines which ports can be used by worker. By default, each node can run four workers on ports 6700, 6701, 6702, and 6703, respectively. For example:
Supervisor. slots. ports:
-6700
-6701
-6702
-6703
Start the last step of Storm and start all background processes of Storm. Like Zookeeper, Storm is also a fail-fast system, so that Storm can be stopped at any time, and the process is correctly resumed after being restarted. This is also why Storm is not saved in the process. Even if Nimbus or Supervisors are restarted, the running Topologies will not be affected.
The following describes how to start various background processes of Storm:

1. Nimbus: Run "bin/Storm nimbus>/dev/null 2> & 1 &" on the storm master node to start the Nimbus background program and run it in the background;
2. Supervisor: Run "bin/Storm supervisor>/dev/null 2> & 1 &" on each storm worker node to start the Supervisor background program and run it in the background;
3. UI: Run "bin/Storm ui>/dev/null 2> & 1 &" on the storm master node to start the UI background program and run it in the background: // {nimbus host}: 8080 observe the cluster's worker resource usage and the running status of Topologies.

Note:

1. After the Storm background process is started, log files of each process will be generated under the logs/subdirectory under the Storm installation and deployment directory.
2. After testing, the Storm UI and Storm Nimbus must be deployed on the same machine; otherwise, the UI will not work properly because the UI process will check whether the native has a Nimbus link.
3. For ease of use, bin/storm can be added to system environment variables.

So far, the Storm cluster has been deployed and configured. You can submit the topology to the cluster for running.
Appendix: main dependencies of Storm:
Apache Zookeeper, Ø MQ, JZMQ, Java 6, and Python 2.6.6