Twitter Storm: Run topology on the Production Cluster

Running topology on the Production cluster is similar to the local mode. The steps are as follows:

1) define topology (if java is used, use TopologyBuilder)

2) use StormSubmitter to submit topology to the cluster. StormSubmitter parameters include the name of topology, the configuration object of topology, And the topology itself.

For example:

12345

Config conf = new Config();conf.setNumWorkers(20);conf.setMaxSpoutPending(5000);StormSubmitter.submitTopology("name",conf, topology);

3) create a jar package that contains your program code and the dependent package that your code depends on (the jar package related to storm is not included, these jar packages will be automatically added to the classpath on the worker node ). If you use maven, the plug-in: Maven Assembly Plugin can help you package, as long as you add the following configuration to your pom. xml.

01020304050607080910111213

<plugin><artifactId>maven-assembly-plugin</artifactId><configuration><descriptorRefs><descriptorRef>jar-with-dependencies</descriptorRef></descriptorRefs><archive><manifest><mainClass>com.path.to.main.Class</mainClass></manifest></archive></configuration></plugin>

Then run mvn assembly: assembly to package it. In other words, they will be automatically added to classpath without the need to include storm-related jar packages.

4) use the storm client to submit the jar package:

1	storm jar allmycode.jar org.me.MyTopology arg1 arg2 arg3

Storm jar submits the code to the cluster and configures the StormSubmitter class to allow it to communicate with the correct cluster. In this example, the storm jar command will call the main function of org. me. MyTopology after uploading the jar package. The parameters are arg1, arg2, and arg3. For more information about how to configure your storm client to communicate with the storm cluster, see configure the storm development environment.

Common configurations

Many topology-level configurations are available. Here we have a list of all configurations. The configuration prefixed with "TOPOLOGY" is a topology-level configuration that can overwrite the global configuration. Below are some common examples:

1) Config. TOPOLOGY_WORKERS: How many worker processes are used to execute the topology. For example, if you set it to 25, there will be a total of 25 java processes in the cluster to execute all the tasks of this topology. If all the components in your topology add up to 150 degree of parallelism, there will be 6 threads (150/25 = 6) in each process ).

2) Config. TOPOLOGY_ACKERS: This configuration sets the number of acker threads. Ackers is part of Storm's reliability API. For more information about storm's reliability API, see how Twitter Storm ensures that messages are not lost.

3) Config. TOPOLOGY_MAX_SPOUT_PENDING: This setting sets the maximum number of unprocessed tuple (no ack/failed) replies on a spout task. We recommend that you set this configuration to prevent the tuple queue from popping up.

4) Config. TOPOLOGY_MESSAGE_TIMEOUT_SECS. The default setting of this setting is 30 seconds, which is sufficient for most topology. For more information about storm reliability APIs, see how Twitter Storm ensures that messages are not lost.

5) Config. TOPOLOGY_SERIALIZATIONS: to use the custom type in your tuple, you can use this configuration to register the custom serializer.

Terminate a topology

To terminate a topology, run the following command:

 

1	storm kill {stormname}

{Stormname} is the name specified when the topology is submitted to the storm cluster.

Storm will not terminate topology immediately. Instead, it terminates all spouts so that they do not launch any new tuple. storm will only kill all worker processes after Config. TOPOLOGY_MESSAGE_TIMEOUT_SECS seconds. This will give topology enough time to complete all the tuple that we have not completed before executing the storm kill command.

Update a running topology

To update a running topology, the only option is to kill the running topology and resubmit a new topology. The command in a plan is to implement a storm swap command to update the topology when running, and ensure that the first and second topology are not running at the same time, and that the replacement causes the least "downtime.

Monitoring topology

The best way to monitor topology is to use Storm UI. Storm UI provides statistics on errors in tasks and the throughput and performance of each component in topology. At the same time, you can view the logs on the worker machine in the cluster.