Storm document (one)----build storm clusters

Reprint Please specify source: http://blog.csdn.net/beitiandijun/article/details/41684717

Source Address: http://storm.apache.org/documentation/Setting-up-a-Storm-cluster.html

This paper describes the construction and operation steps of Storm cluster. If you're going to do it on AWS, you can use the Storm-deploy project. The Storm-deploy automatically downloads, configures, and installs the Storm cluster on the EC2. It also configures the ganglia for you to monitor CPU, disk, and network usage.

If you're having trouble using a storm cluster, you can turn to troubleshooting or send an email to the mailing list.

Here is a summary of the steps to configure the Storm cluster:

1. Configure Storm Cluster

2. Installation of Nimbus and worker machine dependent libraries

3. Download and unzip the storm release version to the Nimbus and worker machines

4, fill in the Storm.yaml must be selected

5. Use the storm script and the supervisor of your choice to start the daemon under supervision

Configuring Zookeeper Clusters

Storm uses zookeeper to coordinate clusters. Zookeeper is not used to transmit messages, so storm is on the zookeeper with low load. A single-node zookeeper cluster should be able to meet most of the situation, but in a large storm cluster, it may require a larger zookeeper cluster, which requires the user to make the tradeoff themselves. Here are the steps to deploy zookeeper.

The following are considerations for zookeeper deployment:

1, point one, you must run zookeeper under Supervison, because zookeeper is a fast failure, so when an error is encountered, it exits the run. See here for more details.

2, point two, configure Cron to compress zookeeper data and transaction log. The zookeeper daemon itself has not done so, if you have not configured Cron,zookeeper speak quickly overflow (out of disk space), see here for more details.

Installing dependent libraries on Nimbus and worker machines

Below, you need to install storm's dependent components on the Nimbus and worker machines.

They are:

1. Java 6

2, Python 2.6.6 (translator Note: This is really a gimmick, I want to install python2.6.6 as required, but found that 2.6.6 can not be completely installed, many modules have been discarded, wasting my time)

These versions have been tested with storm. With different versions, Storm may not work.

Download and pressurize the Storm release version to Nimbus and worker machines

Below, download the Storm release version, then unzip the zip file and place them on Nimbus and on each worker machine. Storm release version download.

Fill in the required options in the Storm.yaml

Storm's configuration file is Storm.yaml under the Conf folder, which configures the daemon for storm. Here is the default configuration. Storm.yaml can overwrite any settings in the Defaults.yaml. Here are some of the required options for getting a working cluster:

1) Storm.zookeeper.servers: This is the host list for the zookeeper cluster serviced by the storm cluster, which is configured as follows:

Storm.zookeeper.servers:

-"111.222.333.444"

-"555.666.777.888"

If the port used by the zookeeper cluster is not the default, you need to set Storm.zookeeper.port.

2) The Storm.local.dir:Nimbus and Supervisor daemons need a directory on the local disk to hold a small amount of state information (such as jars, Confs, and the like). You can create a directory on each machine, set the appropriate permissions, and then populate the directory with the configuration file. For example:

Storm.local.dir: "/mnt/storm"

3) Nimbus.host:worker node needs to know which machine is master, so you can download topology jars and Confs. For example:

Nimbus.host: "111.222.333.444"

4) Supervisor.slots.ports: For each machine, you can use this configuration option to configure how many workers are running on each machine. Each worker uses a separate port to accept the message, and this setting defines which port is used. 5 ports,storm are defined here and 5 works are assigned to run on this machine. If you define 3 ports, Storm will run a maximum of 3 works. By default, it will be set to 4 ports, which is run 4 workers, with the ports: 6700,6701,6702,6703. For example:

Supervisor.slots.ports:

-6700

-6701

-6702

-6703

Use the "Storm" script to run the daemon under supervision and the supervisor you choose

The final step is to run all Storm daemons. The key is to run each daemon under supervision. Storm is a fast-failing system, which means that when any accident occurs, it can cause the process to abort. Storm is designed so that it can stop safely at any time, and then recover correctly when the program restarts. This is why storm remains stateless within the process---if Nimbus or supervisors restarts, running topologies is not affected. Here's how to run the Storm daemon:

1. Nimbus: Run command line under supervision of Master machine: Bin/storm Nimbus

2. Supervisor: Run command line under Supervison on each worker machine: Bin/storm Supervisor. The Supervisor Daemon is responsible for initiating worker processes on his machine.

3. UI: Run the Storm UI through the command line "Bin/storm UI" (You can view the statistics of clusters and topologies on the browser by providing a URL via Storm). The UI can be accessed via a Web browser:

Http://{nimbus hots}:8080

Storm files----build storm clusters