Storm topology and group finishing

Storm's communication mechanism needs to meet some of the following conditions to satisfy storm semantics.

1. Establish a buffer for data transmission. Caches the data sent before the communication connection is established. The data sender can send a message before the connection is established, without having to wait for the connection to be established, but the receiver is running independently.

2, in the message transport layer to ensure that the message can only be sent at most once, the storm system has an ACK mechanism, is not sent successfully message will be re-sent, if the message plane also resend, will cause the message sent multiple times.

This message mechanism is defined by two interfaces,Backtype.storm.messaging.IContext and Backtype.storm.messaging.IConnection.

Icontext is responsible for client and server-side connections, there are four main methods.

1. Prepare (Map stormconf): The Prepare method, which is always defined by storm, can receive the configuration of storm.

2, term (): termination, the method will be called when the worker unloads the transfer plug-in, the custom implementation can be released in the resource occupied here.

3, bind (String topologyid,int port): establishes the server-side connection.

4. Connect (String stormid,string host,int Port): Establish a client connection.

Iconnect defines the interface that sends and receives data on the Icontext.

1, recv (int flag): Receive message.

2, send(int taskid,byte[] payload): Send message.

3, Close (): Call when the connection is closed, release the associated resources.

Topology principle Finishing

From the point of view of the actual execution process of the runtime topology, the job is an instance of multiple components, that is, the task, according to the logical order of the CVS and the concurrency of the configuration, the structure of the flow chart is formed.

　　Stream is the abstraction of data passed in storm, which is a tuple sequence of data items that are infinite in time. Spout is the source of stream, gets data items from a specific data source for topology, and emits (emit) a stream into the job. (Kafkaspout is used in the project, after receiving the data check and then using emit to send to bolt), Bolt can accept any number of upstream delivery stream as input, processing the data, or can be executed after the bolt finishes processing (emit) Launch a new stream to continue processing the downstream bolts.

　　a tuple in a stream can be specified as a structure, consisting of one or more fields (field). the definition of a tuple does not have to be strictly uniform, but it can be defined in each spout,bolt. By default, a tuple can contain basic types, such as integers, longs, shorts, bytes, strings, doubles, floats, booleans, and byte arrays.

Flow Group Mode

1, Shuffle Grouping Random Group

 Public void createtopology (Topologybuilder builder) {    = getkafkaspout (topicname);     // add a spout to the topology     builder.setspout (...)     // add a bolt to the topology, set the degree of parallelism, send it in a random group, and the shufflegrouping parameter is the ID    of the source build Builder.setbolet (Boltname,new Blacklistbolt (), 3). shufflegrouping (Spoutname);
}

In this flow group mode, the source component sends its data items to all of its target components in a random manner, guaranteeing that each target component receives an approximate number of tuples.

2. All Grouping Copy Group

// allgrouping (java.lang.String ComponentID) // allgrouping (java.lang.String componentid,java.lang.string streamid) // The parameter streamid is the identity of the declared stream builder.setbolet (boltname,new Blacklistbolt (), 3). allgrouping (Spoutname, " Signals ");

In this mode, the source component sends the data item it sends to all its target components in the form of a copy, which guarantees that each target component receives the same tuple, just like the Zookeeper profile synchronization, each bolt receives the same copy.

3, Global Grouping Group

In this mode, the source component sends all the data items it sends to an instance of the target component, which is the task with the smallest ID in the component. Ensures that all data items are processed only by one instance of the target component (a bolt)

Builder.setbolet (Boltname,new Blacklistbolt (), 3). globalgrouping (Spoutname);

4.Fiellds Grouping grouped by field

Builder.setbolet (Boltname,new Blacklistbolt (), 3). fieldsgrouping (Spoutname,new Field ("domain name");

The source component sends its data items, grouped by the values of the specified fields in a tuple, to the downstream target component, which guarantees that a tuple with values of the same domain combination is sent to the same bolt.

5, direct Grouping directly group

Builder.setspout ("Kafkaspout", Topicspout) Builder.setbolt (boltname1,new boltName1 (), 1). Shufflegrouping ("Kafkaspout"); // The data item Builder.setbolt (boltname2,new boltname2 (), 2) sent by the above bolts is received in a direct packet mode . directgrouping (BOLTNAME1);

The source component sends the data item it sends, directly specifying the target component, and enables the specified component to receive a given tuple. It should be noted that the Executle () function of the receiving bolt, yo ah use emitdirect () instead of emit, Used to send data items to a designated celebrity

Build topology

There are three types of methods for building Topologybuilder: creating topology, adding bolts, and adding spout methods. The Setbolt and Setspout interfaces each have different overloaded methods, returning the object used to declare the input of the component.

1,ID: The identity of the component (spout, Bolt), the string type, if you need to reference the component, use the identity ID specified here. For example, using "Kafkaspout"

2,Bolt: Add Bolt object, and then Setbolt overloaded method, there are two types of Irichbolt and Ibasicbolt bolt parameters, the project is used in Irichbolt, the difference is that Basicbolt for non-clustered processing , capable of automatic (anchoring) and (acking)

3.spout: Added spout object, in Setspout method, this parameter is the spout interface of the irichspout type.

4,Parallelism_hint: Degree of parallelism, numerical parameters. Sets the number of threads that will be allocated when the component is run.

Reference: "Storm Big Data streaming computing and application practice"

Storm topology and group finishing