Apache Storm Series II Storm-0.9.0.1 version installation deployment

After Twitter Storm updated 0.9.0.1, installing deployment became much easier, compared to the storm0.8.x version, where Storm had fewer zeromq and JZMQ installations, and saved a lot of bugs when compiling the plugins. 1, Storm-0.9.0.1 version of the highlights:

1.1, Netty Transport

The first highlight of Storm 0.9.0.1 was the introduction of the Netty transport. The storm network transmission mechanism realizes the pluggable form, currently contains two ways: the original 0MQ transmission, as well as the new Netty implementation; In earlier versions (prior to 0.9.x), Storm only supported 0MQ transmissions, since 0MQ was a local library (native), The dependency on the platform is high, and it is challenging to install it completely correctly. And the version of the difference between the larger; Netty Transport provides a pure Java alternative, eliminating storm's local library dependency, and more than 0MQ of network transmission performance more than one times faster;

add: Before writing this document to forget the configuration Netty transport way to write in, so many friends can be configured after the background will be reported an error, this is still used in the original way, so the following add the way storm uses Netty configuration;

to configure Storm to use Netty transport need to add the following configuration and adjust values to your Storm.yaml file, the response configuration values can be adjusted according to your needs

Storm.messaging.transport: "Backtype.storm.messaging.netty.Context"--Specify the transport protocol
Storm.messaging.netty.server_ Worker_threads:1   --Specifies the number of Netty server worker threads
storm.messaging.netty.client_worker_threads:1   -- Specify the number of Netty client worker threads
storm.messaging.netty.buffer_size:5242880    --Specify the Netty cache size
Storm.messaging.netty.max _retries:100   --Specify maximum number of retries
storm.messaging.netty.max_wait_ms:1000  --Specify maximum wait time (milliseconds)
storm.messaging.netty.min_wait_ms:100   --Specify the minimum wait time (milliseconds)

1.2. Log modification

Storm 0.9. Version 0.1 provides new features to help debug and detect topologies: logviewer (daemon name)

You can now view the corresponding logbook in the Storm UI by clicking on the appropriate woker. A bit similar to the kind of Hadoop log viewing mechanism.

1.3. Security

Security, certification, and authorization have been and will continue to be key areas of focus for the future. Storm 0.9. Version 0.1 introduces a pluggable serialization tuple API and a use case for encrypting sensitive data based on the Blowfish encryption method.

The main improvement is the above three points. The others will not mention it. 2. Storm-0.9.0.1 Installation and Deployment

This section will describe in detail how to build a storm cluster. Here are the installation steps that need to be completed sequentially

1. Build zookeeper cluster;

2. Dependent Library Installation

3. Download and unzip the storm release version;

4. Modify the Storm.yaml configuration file;

5. Start storm's various background processes. 2.1 Building Zookeeper clusters

Storm uses zookeeper to coordinate the cluster, and because zookeeper is not used for messaging, the pressure that storm brings to zookeeper is rather low. In most cases, the zookeeper cluster of a single node is adequate, but to ensure failure recovery or deployment of a large storm cluster, the zookeeper cluster of larger nodes may be required (3 of the official recommended minimum number of nodes for a zookeeper cluster). Complete the following installation deployment steps on each machine in the Zookeeper cluster:

1. Download and install Java JDK, the official download link is Http://java.sun.com/javase/downloads/index.jsp,JDK version of JDK 6 or above.

2. According to the load situation of zookeeper cluster, reasonable set Java heap size, avoid swap as far as possible, cause zookeeper performance to degrade. For the sake of conservatism, a machine with 4GB of memory can allocate 3GB maximum heap space for zookeeper.

3. After download, unzip the installation zookeeper package, the official download link is http://hadoop.apache.org/zookeeper/releases.html.

4. Based on the Zookeeper cluster node, create the Zookeeper configuration file zoo.cfg in the Conf directory:

ticktime=2000
datadir=/var/zookeeper/
clientport=2181
initlimit=5
synclimit=2
server.1= zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888

5. Create the myID file in the DataDir directory, which contains only one row, and the content is the ID number in the server.id corresponding to that node. Where datadir specifies the zookeeper data file directory, where Server.id=host:port:port,id is the number of each zookeeper node, saved in DataDir file in myID directory, zookeeper1~ Zookeeper3 represents the hostname of each zookeeper node, the first port is the one used to connect leader, and the second port is the one used for leader elections.

6. Start the Zookeeper service:

bin/zkserver.sh start

7. Whether the service is available through Zookeeper client testing:

Bin/zkcli.sh-server 127.0.0.1:2181

2.2 Dependent Library installation

The storm-dependent libraries here are python, and JDK two, which are relatively simple to install, so don't mention it here. 2.3 Download and unzip the storm release version

The Storm0.9.0.1 version provides two forms of compression packages: Zip and tar.gz

We download the tar.gz format, which eliminates the uzip installation

Download path: https://dl.dropboxusercontent.com/s/tqdpoif32gufapo/storm-0.9.0.1.tar.gz

Unzip command

TAR-ZXVF storm-0.9.0.1.tar.gz

2.4 Download and unzip the storm release version

The storm release version has a Conf/storm.yaml file under the Extract directory to configure Storm. The default configuration can be viewed here. The configuration option in Conf/storm.yaml overrides the default configuration in Defaults.yaml. The following configuration options are required to be configured in Conf/storm.yaml:

1) The Zookeeper cluster address used by the Storm.zookeeper.servers:Storm cluster, in the following format:

Storm.zookeeper.servers:
-"111.222.333.444″
-" 555.666.777.888″

If the zookeeper cluster is not using the default port, then the Storm.zookeeper.port option is also required.

2) The Storm.local.dir:Nimbus and supervisor processes are used to store a small amount of state, such as the local disk directory of Jars, Confs, and so on, and need to be created in advance and given sufficient access rights. The directory is then configured in Storm.yaml, such as:

Storm.local.dir: "/home/admin/storm/workdir"

3) Nimbus.host:Storm cluster Nimbus machine address, each supervisor working node need to know which machine is Nimbus, in order to download topologies jars, Confs and other files, such as:

Nimbus.host: "111.222.333.444"

4) Supervisor.slots.ports: For each Supervisor work node, you need to configure the number of workers that the worker node can run. Each worker occupies a separate port for receiving messages, and the configuration option is used to define which ports are available for use by the worker. By default, 4 workers can be run on each node, at 6700, 6701, 6702, and 6703 ports, such as:

Supervisor.slots.ports:
    -6700
    -6701
    -6702
    -6703

2.5 Start storm various background processes

The final step is to start all background processes for storm. Like zookeeper, Storm is also a fast-failing (fail-fast) system, so that storm can be stopped at any time and executed correctly when the process restarts. This is why storm is not saving state in the process, and even if Nimbus or supervisors are restarted, the running topologies will not be affected.
Here's how to start storm's various background processes:
Nimbus: Run "bin/storm Nimbus >/dev/null 2>&1 &" On the Storm master node to launch the Nimbus daemon, and put it into the background execution;
Supervisor: Run "bin/storm supervisor>/dev/null 2>&1 &" On the Storm's various work nodes to start the Supervisor daemon, and put it into the background execution;
UI: Run "Bin/storm UI >/dev/null 2>&1 &" On the Storm master node to start the UI daemon and put it in the background, which can be done via http://{nimbushost after startup. : 8080 Observe the usage of worker resources in the cluster, the running status of topologies, and other information.

LogView: Run "bin/storm logviewer >/dev/null 2>&1" On the Storm master node to launch the Logviewer daemon and put it in the background.

Precautions:
When you start the storm background process, you need to have write access to the Storm.local.dir directory that is set in the Conf/storm.yaml configuration file.
After the storm daemon process is started, the log files for each process are generated under the logs/subdirectory under the Storm installation deployment directory.
After testing, the storm UI must be deployed on the same machine as the Storm Nimbus, otherwise the UI will not work because the UI process checks to see if there is a Nimbus link on the machine.
For ease of use, bin/storm can be added to the system environment variable.
Now that the storm cluster has been deployed and configured, you can submit the topology to the cluster to run.

Next we check how the Environment works:--use JPS to check daemon health

zqgame@kickseed:/data/storm/zookeeper-3.4.5/bin$ JPS
20420 nimbus
20623 logviewer
20486 Supervisor
20319 Core
21755 Jps

View the run page as follows

Reference documents:

Http://storm-project.net/2013/12/08/storm090-released.html

http://blog.linezing.com/2013/01/how-to-install-and-deploy-storm-cluster#comment-9070--storm-0.8.x version can refer to this document installation