Storm 1.0.0

Storm 1.0. Version 0 adds a lot of new features, usability and performance improvements, which are a milestone version of Storm development, with the following key features.

Performance improvements

Storm 1.0. The biggest highlight of the 0 version is the performance boost, which can be increased to 16 times times faster than the previous version, and the delay can be reduced to 60%. The topology performance of storm is related to application cases and dependent external services, but for most applications, performance can be up to 3 times times higher than in previous versions.

pacemaker-Heartbeat Server

Pacemaker is an optional background process in storm to handle the worker heartbeat. When the storm cluster is large, all the workers send the heartbeat to zookeeper, because the data on the zookeeper is write disk, and in order to achieve the consistency of the data, the leader node and the follow node in the zookeeper to communicate, Also comes with a lot of network traffic overhead, so zookeeper is easy to become a performance bottleneck.
Because the heartbeat data is generally temporary, you do not need to persist it to your hard disk, and you do not need to synchronize data across nodes. Store the heartbeat data in memory on the line, pacemaker is the main thing to do is this function. Pacemaker provides a simple memory-based key/value store, where the storage pattern is similar to Zookeeper,key maintained in the form of a directory, where value is the byte data.

Distributed Cache API

In previous releases, storm developers generally wrapped the resources needed for a topology (such as query data, machine learning models) and SWAT into a topology jar package. The problem with this implementation is the difficulty of updating, and if you want to update the resources on which the topology depends, you have to repackage and deploy. Another issue, if the dependent data is large (GB or larger), this can greatly increase the time-to-start of the topology.
The Storm 1.0 release uses the distributed cache API to implement file (BLOBs) sharing across multiple topologies. Files in the distributed cache can be updated from the command line without redeploying the topology. The file size in the distributed cache can be a few kilobytes, or several gigabytes, and it also supports ZIP and gzip compression formats.
Storm 1.0 supports two ways of implementing the distributed Cache API. One is based on the local file system on the Supervisor node, and the other is based on the HDFS implementation. Both implementations support fine-grained ACL access control.

Nimbus HA

In previous versions of Storm, the Nimbus node had a single point of failure (the Nimbus node would not affect the running topology), but if the Nimbus node did not exist, the user could not commit the new topology and the task of the previous topology could not be redistributed.
In Storm 1.0, the HA Nimbus was used to solve the single point of failure. Running multiple Nimbus service instances in the cluster, when the Nimbus node is hung up, the new Nimubs node is re-elected, and the Nimbus host can join or leave the cluster at any time. HA Nimbus ensures the availability of the topology resources by taking the distributed cache API to make backup of the data.

Native Streaming window API

Window-based computing is very common in stream processing, and continuous data streams can be divided into discrete batches of data through specific criteria such as time, which can be calculated separately for each batch of data. A typical example is the calculation of the most popular Twitter themes in the past hour.
window calculations can be used to implement aggregations, joins, pattern matching, and so on. A window can be seen as a memory-based table that can be added to a table or deleted from a table based on a certain policy (such as time).
In previous versions, Storm developers needed to build their own window calculation logic, missing some high-level abstractions, and based on this high-level abstraction, the user could define windows in a standard way in a topology.
The native streaming window API is available in Storm 1.0, and the window definition consists of two parameters: window length and window sliding interval. Storm supports sliding windows and scrolling windows two ways, the window size can be based on the length of time or the number of events.

State management-Automatic checkpoint of stateful bolts

Storm 1.0 introduces stateful Bolt APIs and supports automatic checkpoint. Stateful bolts are easy to implement, just inherit the Basestatefulbolt class, and in the topology, stateful bolts and stateless bolts can be used together. Storm can automatically manage the status of bolts, such as automatic checkpoint, and when a failure occurs, Storm can restore the state of the bolt.
Storm 1.0 can manage the state through memory and Redis, and later in the release, consider adding additional state storage methods.

Automatic back pressure mechanism

In previous releases, the way to limit the data traffic injected into the topology was to enable the acking mechanism and set the topology.max.spout.pending parameter. This approach can greatly degrade performance when the use case does not need to implement at-least-once semantic fault tolerance.
Storm 1.0 introduces an automatic back-pressure mechanism based on high/low water levels, where the water level is represented by the buffer size of the task. When the buffer reaches the high water level, the anti-pressure mechanism automatically triggers, reducing the spout data injection rate until the low water level is reached.
Storm's anti-pressure mechanism and the spout API are independent, so all existing spout support automatic back-pressure.

Resource-Aware Scheduler

Storm supports a pluggable topology scheduler, and Storm 1.0 provides a resource-based scheduler that takes into account the memory in the cluster (both in-heap and out-of-heap) and CPU resources. The resource-aware scheduler (RAS) allows the user to specify the required memory and CPU resources for the topology component (Spout/bolt), and Storm dispatches the topology task between different workers to best meet the resource requirements of those tasks.
In the future, the storm community will extend the RAS implementation, taking into account network resource overhead and rack awareness.

Dynamic Log Level

Storm 1.0 allows users and administrators to dynamically adjust the log level of a running topology, which can be implemented through the Storm UI or the command line, and the user can configure an optional timeout period, which is automatically restored once the timeout is exceeded. Log files can be found through the Storm UI or the Logviewer service.

Tuple sampling and commissioning

During the debugging of the topology, many storm users take additional debug bolts or Trident functions to record the data flow information in the topology, and Storm 1.0 provides a new topology debugging capability.
The Storm UI provides a feature that allows users to scale samples of tuples that flow into a topology or a specific component, which can be either observed directly from the storm UI or stored on a hard disk.

Distributed log Lookups

Another feature added by the Storm UI is distributed log lookups, where the lookup object can be all log files for a particular topology, and the lookup results contain the matching results for all supervisor nodes.

Dynamic analysis of worker performance

Another feature enhancement is the dynamic worker performance analysis, which allows the user to obtain analysis data for the worker through the storm UI, including:
-Heap Dumps
-Jstack Output
-Jprofile Records
These analysis data can be downloaded directly, used for offline analysis, and workers can be restarted via the Storm UI.
Original Digest from http://blog.csdn.net/wfzczangpeng/article/details/52711296

Storm 1.0.0