Storm (ii): Cluster deployment Configuration

First, Introduction

Apache Storm distributed cluster main node by the control node (Nimbus node) and the working node (Supervisor node), the control node can be one, the working node is composed of multiple, The zookeeper is mainly responsible for the coordination between Nimbus node and Supervisor node.

Nimbus:nimbus node is responsible for resource allocation and task assignment we run the topology, submit the topology through the command storm Jar, commit to the Nimbus,nimbus distribution task to supervisor, and monitor the supervisor State, If the supervisor hangs, the task is re-assigned to the other supervisor.

The Supervisor:supervisor node periodically accepts the Nimbus node assignment task and downloads the code from Nimbus, initiates the corresponding worker process, and monitors the worker's heartbeat, If a worker hangs out due to a memory overflow or other cause, supervisor restarts the worker.

The Storm UI is the monitoring page of the storm cluster, where you can view information such as Nimbus, supervisor distribution, and status to help us manage storm clusters and topologies.

Second, Storm cluster deployment

1, two virtual machines: 192.168.2.200, 192.168.2.201

1) Storm Download address: http://storm.apache.org/downloads.html, I downloaded version apache-storm-1.1.0.tar.gz

2) Unzip the two virtual machines separately

          > TAR-XZF apache-storm-1.1.0.tar.gz
          > CD apache-storm-1.1.0/conf

2. Configuring the Conf/storm.yaml file

1) Configure the address of the connection zookeeper, the default port is 2181, you can modify the corresponding zookeeper port

        Storm.zookeeper.port
        storm.zookeeper.servers:
           -"Master"
           -"Salve2"

2) Configure the address of the Nimbus

        Nimbus.seeds: ["Master"]

3) configuration Nimbus

Configure the listening port used by each worker on the Supervisor node, each worker has a separate listening port, and a supervisor default of 4 ports.

        Supervisor.slots.ports:
          -6700
          -6701
          -6702
          -6703

4) Configuring the UI port

        ui.port:8081

3, Storm's state information (Nimbus distributed tasks, Supervisor, worker's heartbeat, etc.) are stored on the zookeeper, through the zookeeper Distributed system coordination to ensure the stability of the storm cluster. Zookeeper the installation configuration, and start the zookeeper.

4. The zookeeper cluster starts first, then starts the 192.168.2.200, 192.168.2.201 the Storm's UI, Nimbus, and supervisor on the virtual machine.

    >bin/storm Nimbus >/dev/null 2>&1 &
    >bin/storm supervisor >/dev/null 2>&1 &
    >bin/storm UI >/dev/null 2>&1 &

We visited the Storm management page HTTP://IP: The Port/index.html,storm UI is the monitoring page for the storm cluster, where you can view information such as Nimbus, supervisor distribution, and status to help us manage Storm clusters and topologies

Iii. issues

Org.apache.storm.utils.NimbusLeaderNotFoundException:Could not find leader Nimbus from seed hosts ["192.168.2.200"]. Did you specify a valid list of Nimbus hosts for config nimbus.seeds?

Delete the zookeeper corresponding storm information and restart zookeeper.

First: Log in first zookeeper

Step two: Delete the corresponding storm