Storm cluster deployment and configuration process detailed

Reprint Address: http://www.cnblogs.com/super-d2/p/5489229.html

Let's take a holistic look at the steps to build a storm cluster: Setting up a zookeeper cluster installation relies on all Nimbus and worker nodes to download and decompress the storm release version to all Nimbus and worker node configurations Storm.yaml start related background processes1 Configuring the Zookeeper clusterWe know that Storm uses zookeeper to coordinate the entire cluster. Zookeeper is not used for message delivery, so storm does not put a lot of pressure on zookeeper. Single-node zookeeper can be a good fit in most situations, but if you want to get better reliability or deploy a large cluster, you might need a large zookeeper cluster.      About the deployment of zookeeper here is not much to do, please refer to: here. About the deployment of zookeeper add two points here: it is critical to run zookeeper under a monitoring process, because zookeeper is a fail-fast process that automatically exits when it encounters any errors, please refer to: here Timing to compress and transfer zookeeper data is also critical, because Zookeeper does not have a compression and clear data mechanism, if we do not set up a cron to manage this data, zookeeper generated data will quickly fill the disk, please refer to: here PS: If zookeeper fails to start, look at the Zookeeper.out file in its bin directory, and configure its myid to try.2 installation relies on Nimbus and worker nodesWhat storm needs to rely on is that Java 6 Python 2.6.6 need to be aware that storm has tested most versions of dependencies, but Storm does not guarantee that dependencies on any version will work correctly.3 Downloads Extract storm release version to Nimbus and worker nodesThe next step is to download the extract Storm compressed file to each machine, and the Storm release version can be downloaded from here.4 Configuring the Storm.yaml file      storm has a file Conf/storm.yaml, which is the storm configuration file. All the default values for this file can be obtained here. The configuration in the Storm.yaml overrides the configuration in the Default.yaml. Configure the configuration that a cluster must modify below:        1) Storm.zookeeper.servers: Configure the list of zookeeper clusters storm.zookeeper.servers:          -  "111.222.333.444"           -  "555.666.777.888"          If the port used by your zookeeper cluster is not the default port, Then you have to configure Storm.zookeeper.port.      2) Storm.local.dir:storm's Nimbus and work processes require a directory to hold a small subset of state data, such as jars, Confs, and so on. We need to create this directory on each machine and give it the appropriate permissions. storm.local.dir:  "/mnt/storm"         3) Nimbus.host:worker node needs to know which machine is the master node, So that you can download jars and confs from the maser node. nimbus.host:  "111.222.333.44"         4) Supervisor.slots.ports: For each worker machine, It determines how many worker processes this machine can run altogether. Each worker process takes a single port to receive the message, which is the configuration of which ports are assigned to the worker process. If you configure 5 ports here, Storm will be able to allocate 5 worker processes to this machine, and if you configure 3 ports, Storm can only allocate 3 worker processes. Storm defaults to allocate 4 worker processes to 6700,6701,6702,6703 Port. For example: supervisor.slots.ports:      - 6700      - 6701       - 6702      - 6703  5 running storm-related daemons with the Storm commandThe final step is to start all storm-related daemons. Of course, it is necessary to include these processes under the supervision process. Storm is also a fail-fast system, which means that these processes terminate as soon as they encounter an exception. Storm is designed so that it can safely terminate at any time and resume when the process restarts. This is why storm does not save the related state in the process, and if the Nimbus or supervisor node restarts, the running topoloies will not be affected. Here's the command to start the storm-related process: Nimbus: Run "Bin/storm Nimbus" on the master node Supervisor: Run "Bin/storm Supervisor" on each worker node, The supervisor process is responsible for starting and stopping the corresponding worker process UI on the worker node: running the Bin/storm UI, a tool for managing and demonstrating the running state of the storm cluster through the page, through the Http://nimbus host          : 8080 "to access. We can see that starting the Storm service process is fairly straightforward, the log generated by storm is stored in the Storm/logs directory of each machine, and Storm manages its logs through Logback. We can change its log directory and content by modifying its logback.xml file.