Storm Cluster related information

1. Storm Cluster components

The storm cluster contains two types of nodes: the master node and the work node. The respective roles are as follows:

• The master node runs a daemon called Nimbus , which distributes the code within the Storm cluster, assigns tasks to the work machine, and monitors the cluster's operational status and monitors for failures . The role of Nimbus is similar to the Jobtracker role in Hadoop.
• A background program called Supervisor is run on each work node. The supervisor is responsible for monitoring the tasks assigned to it from Nimbus to start or stop the worker process that executes the task. Each worker process executes a subset of topology; A running topology consists of multiple worker processes that are distributed across different worker nodes.

Storm cluster Components

all coordination between the Nimbus and supervisor nodes is achieved through the zookeeper cluster . In addition, both the Nimbus and supervisor processes are fast-failing (fail-fast) and stateless (stateless), and all the state of the storm cluster is either in a zookeeper cluster or on a local disk. This means that you can use kill-9 to kill Nimbus and supervisor processes, which can continue to work after a reboot. This design allows the storm cluster to have incredible stability.

Storm's fault tolerance is divided into the following types. (1) Worker failure in worker process: if the worker process of a node "dies", the supervisor process attempts to restart the worker. If a continuous restart worker fails or the worker cannot periodically report a "heartbeat" to Nimbus, Nimbus assigns the task to other nodes on the cluster to execute. (2) cluster node failure: If a node in the cluster fails, all tasks assigned to that node will fail due to timeouts, and Nimbus will reassign all tasks assigned to that node to the other nodes in the cluster. (3) Nimbus or supervisor Daemon failed: Both Nimbus and supervisor are designed to fail quickly (fail quickly when encountering an unknown error) and stateless (all state information is stored on zookeeper or on disk). The Nimbus and Supervisor Daemons must run with the help of some monitoring tools (for example, Daemontools or monitor), and once the Nimbus or supervisor fail, they can be restarted immediately, and the entire cluster is as if nothing has happened. Most importantly, no worker process is affected by the failure of Nimbus or supervisor, and this feature of Storm is in stark contrast to Hadoop, and if the jobtracker fails, all tasks will fail. (4) Nimbus node failure: If the node where the Nimbus daemon resides fails, worker process workers on the worker node continue to perform the compute task, and if the worker process fails, the supervisor process restarts the failed worker task on that node. However, when there is no nimbus effect, all worker tasks are not assigned to other worker nodes on the machine, even if the worker is on the machine that is failing. 2. Install Storm cluster

Here are the installation steps that need to be done sequentially:

• Build zookeeper cluster;
• Install storm-dependent libraries;
• Download and unzip the storm release version;
• Modify the Storm.yaml configuration file;
• Start storm each background process.

2.1 Build Zookeeper cluster this blog is already available (no longer described here )

Precautions:

1. Since zookeeper is a fast-failing (fail-fast) and encounters any error conditions, the process exits, so it is best to manage the zookeeper through a monitoring program to ensure that the zookeeper can be restarted automatically after exiting.
2. Zookeeper runs a lot of log and snapshot files in the DataDir directory, and the zookeeper run process is not responsible for periodically cleaning up and merging these files, resulting in large disk space, so you need to periodically purge unused logs and snapshot files through cron.

2.2 Installing storm-dependent libraries

Next, Storm's dependent libraries need to be installed on the Nimbus and supervisor machines, as follows:

• ZeroMQ 2.1.7
• Jzmq
• Java 6
• Python 2.6.6
• Unzip

2.2.1 Installing ZEROMQ 2.1.7:

wget HTTP://DOWNLOAD.ZEROMQ.ORG/ZEROMQ-2.1.7.TAR.GZTAR-XZF ZEROMQ-2.1.7.TAR.GZCD Zeromq-2.1.7./configuremakemake Install

Precautions: 1. If the installation process does not find an error uuid, install the UUID library with the following package:

Yum Install E2fsprogs-devel

2.2.2 Installing JZMQ:

To ensure that the JZMQ is working properly, you may need to complete the following configurations:

1. Set Java_home environment variables correctly
2. Installing the Java Development Package
3. Upgrade autoconf

wget https://github.com/zeromq/jzmq/archive/master.zipunzip master.zipcd jzmq-  Master. /autogen.sh. /Configuremakemake Install

Check the installation of ZEROMQ and JZMQ:

See if the libraries for ZEROMQ and JZMQ are already in place, typically in the/usr/local/lib directory:

Ls-l/usr/local/lib

2.2.3 Installing Java 6

2.2.4 Installation Python2.6.6

2.2.5 installing unzip, Automake

2.3 Download and unzip the storm release version

1. To download the storm release, it is recommended to use Storm0.8.2:

Storm Cluster related information