Storm-0.9.0.1 Installation Deployment Guide

to read this article with the following questions:
1.Storm only supports what transport
2. What configuration can be made to change the zookeeper default port
3.Storm UI must be deployed on the same machine as the Storm Nimbus, the UI does not work why


2.storm-0.9.0.1 about several steps

Twitter After storm update 0.9.0.1, installing the deployment became much easier than the storm0.8.x version, where Storm had fewer zeromq and JZMQ installations and a lot of bugs when compiling the plugins. The highlights of the
1, Storm-0.9.0.1 release:
1.1, Netty Transport
Storm 0.9.0.1 the first highlight of the release is the introduction of Netty Transport. The storm network transmission mechanism realizes the pluggable form, currently contains two ways: the original 0MQ transmission, as well as the new Netty implementation; In earlier versions (prior to 0.9.x), Storm only supported 0MQ transmissions, since 0MQ was a local library (native), The dependency on the platform is high, and it is challenging to install it completely correctly. And the version of the difference between the larger; Netty Transport provides a pure Java alternative, eliminating storm's local library dependency and more than 0MQ of network transmission performance more than one time;
Add: Forget to write the configuration before writing this document Netty Transport way to write in, so maybe a lot of friends configure the background will be reported an error, this is still used in the original way, so below to add the way storm uses Netty configuration;



To configure storm to use Netty transport need to add the following configuration and adjust values to your Storm.yaml file, the response configuration values can be adjusted according to your needs

1. Storm.messaging.transport: "Backtype.storm.messaging.netty.Context"--Specify the transport protocol
   
2. Storm.messaging.netty.server_worker_threads:1--Specify the number of Netty server worker threads
   
3. Storm.messaging.netty.client_worker_threads:1--Specify the number of Netty client worker threads
   
4. storm.messaging.netty.buffer_size:5242880--Specify Netty cache size
   
5. STORM.MESSAGING.NETTY.MAX_RETRIES:100--Specify the maximum number of retries
   
6. storm.messaging.netty.max_wait_ms:1000--Specify maximum wait time (milliseconds)
   
7. STORM.MESSAGING.NETTY.MIN_WAIT_MS:100--Specify the minimum wait time (milliseconds)

Copy Code

1.2. Log modification
Storm 0.9. Version 0.1 provides new features to help debug and detect topologies: logviewer (daemon name)
You can now view the corresponding logbook in the Storm UI by clicking on the appropriate woker. A bit similar to the kind of Hadoop log viewing mechanism.
1.3. Security
Security, certification, and authorization have been and will continue to be key areas of focus for the future. Storm 0.9. Version 0.1 introduces a pluggable serialization tuple API and a use case for encrypting sensitive data based on the Blowfish encryption method.
The main improvement is the above three points. The others don't mention it!
2. Storm-0.9.0.1 Installation and Deployment

This section will describe in detail how to build a storm cluster. Here are the installation steps that need to be completed sequentially
1. Build zookeeper cluster;
2. Dependent Library Installation
3. Download and unzip the storm release version;
4. Modify the Storm.yaml configuration file;
5. Start storm's various background processes.

2.1 Building Zookeeper clusters

Storm uses zookeeper to coordinate the cluster, and because zookeeper is not used for messaging, the pressure that storm brings to zookeeper is rather low. In most cases, the zookeeper cluster of a single node is adequate, but to ensure failure recovery or deployment of a large storm cluster, the zookeeper cluster of larger nodes may be required (3 of the official recommended minimum number of nodes for a zookeeper cluster). Complete the following installation deployment steps on each machine in the Zookeeper cluster:
1. Download and install Java JDK, the official download link is Http://java.sun.com/javase/downloads/index.jsp,JDK version of JDK 6 or above.
2. According to the load situation of zookeeper cluster, reasonable set Java heap size, avoid swap as far as possible, cause zookeeper performance to degrade. For the sake of conservatism, a machine with 4GB of memory can allocate 3GB maximum heap space for zookeeper.
3. After download, unzip the installation zookeeper package, the official download link is http://hadoop.apache.org/zookeeper/releases.html.
4. Based on the Zookeeper cluster node, create the Zookeeper configuration file zoo.cfg in the Conf directory:

1. ticktime=2000
   
2. datadir=/var/zookeeper/
   
3. clientport=2181
   
4. Initlimit=5
   
5. synclimit=2
   
6. server.1=zookeeper1:2888:3888
   
7. server.2=zookeeper2:2888:3888
   
8. server.3=zookeeper3:2888:3888

Copy Code

5. Create the myID file in the DataDir directory, which contains only one row, and the content is the ID number in the server.id corresponding to that node. Where datadir specifies the zookeeper data file directory, where Server.id=host:port:port,id is the number of each zookeeper node, saved in DataDir file in myID directory, zookeeper1~ Zookeeper3 represents the hostname of each zookeeper node, the first port is the one used to connect leader, and the second port is the one used for leader elections.
6. Start the Zookeeper service:

1. bin/zkserver.sh start

Copy Code

7. Whether the service is available through Zookeeper client testing:

1. Bin/zkcli.sh-server 127.0.0.1:2181

Copy Code

2.2 Dependent Library Installation

The storm-dependent libraries here are python, and JDK two, which are relatively simple to install, so don't mention it here!

2.3 Download and unzip the storm release version

The Storm0.9.0.1 version provides two forms of compression packages: Zip and tar.gz
We download the tar.gz format, which eliminates the uzip installation
Download path: https://dl.dropboxusercontent.co ... torm-0.9.0.1.tar.gz
Unzip command

1. TAR-ZXVF storm-0.9.0.1.tar.gz

Copy Code

2.4 Download and unzip the storm release version

The storm release version has a Conf/storm.yaml file under the Extract directory to configure Storm. The default configuration can be viewed here. The configuration option in Conf/storm.yaml overrides the default configuration in Defaults.yaml. The following configuration options are required to be configured in Conf/storm.yaml:
1) The Zookeeper cluster address used by the Storm.zookeeper.servers:Storm cluster, in the following format:

1. Storm.zookeeper.servers:
   
2. -"111.222.333.444″
   
3. -"555.666.777.888″

Copy Code

If the zookeeper cluster is not using the default port, then the Storm.zookeeper.port option is also required.

2) The Storm.local.dir:Nimbus and supervisor processes are used to store a small amount of state, such as the local disk directory of Jars, Confs, and so on, and need to be created in advance and given sufficient access rights. The directory is then configured in Storm.yaml, such as:

1. Storm.local.dir: "/home/admin/storm/workdir"

Copy Code

3) Nimbus.host:Storm cluster Nimbus machine address, each supervisor working node need to know which machine is Nimbus, in order to download topologies jars, Confs and other files, such as:

1. 01.nimbus.host: "111.222.333.444"

Copy Code

4) Supervisor.slots.ports: For each Supervisor work node, you need to configure the number of workers that the worker node can run. Each worker occupies a separate port for receiving messages, and the configuration option is used to define which ports are available for use by the worker. By default, 4 workers can be run on each node, at 6700, 6701, 6702, and 6703 ports, such as:

1. Supervisor.slots.ports:
   
2. -6700
   
3. -6701
   
4. -6702
   
5. -6703

Copy Code

2.5 Start storm various background processes

The final step is to start all background processes for storm. Like zookeeper, Storm is also a fast-failing (fail-fast) system, so that storm can be stopped at any time and executed correctly when the process restarts. This is why storm is not saving state in the process, and even if Nimbus or supervisors are restarted, the running topologies will not be affected.
Here's how to start storm's various background processes:
Nimbus: Run "bin/storm Nimbus >/dev/null 2>&1 &" On the Storm master node to launch the Nimbus daemon, and put it into the background execution;
Supervisor: Run "bin/storm supervisor>/dev/null 2>&1 &" On the Storm's various work nodes to start the Supervisor daemon, and put it into the background execution;
UI: Run "Bin/storm UI >/dev/null 2>&1 &" On the Storm master node to start the UI daemon and put it in the background, which can be done via http://{nimbushost after startup. : 8080 Observe the usage of worker resources in the cluster, the running status of topologies, and other information.
LogView: Run "bin/storm logviewer >/dev/null 2>&1" On the Storm master node to launch the Logviewer daemon and put it in the background.
Precautions:
When you start the storm background process, you need to have write access to the Storm.local.dir directory that is set in the Conf/storm.yaml configuration file.
After the storm daemon process is started, the log files for each process are generated under the logs/subdirectory under the Storm installation deployment directory.
After testing, the storm UI must be deployed on the same machine as the Storm Nimbus, otherwise the UI will not work because the UI process checks to see if there is a Nimbus link on the machine.
For ease of use, bin/storm can be added to the system environment variable.
Now that the storm cluster has been deployed and configured, you can submit the topology to the cluster to run.

Next we check how the Environment works:--use JPS to check daemon health

1. [Email protected]:/data/storm/zookeeper-3.4.5/bin$ JPS
   
2. 20420 Nimbus
   
3. 20623 Logviewer
   
4. 20486 Supervisor
   
5. 20319 Core
   
6. 21755 Jps
   

Copy Code

View the run page as follows

Storm-0.9.0.1 Installation Deployment Guide