Kafka Actual Case Analysis Summary __kafka

PrefaceThe basic features and concepts of Kafka are introduced. This paper introduces the selection of MQ, the practical application and the production monitoring skill of Kafka in combination with the application requirement design scene. introduction of main characteristics of Kafka

Kafka is a distributed,partitioned,replicated commit logservice. It provides a JMS-like feature, but it is completely different in design implementations, and it is not an implementation of the JMS specification. Kafka classifies messages according to topic, sends the message to producer, the message recipient becomes consumer, and the Kafka cluster has multiple Kafka instances, each instance (server) becomes broker. Both the Kafka cluster and the producer and consumer rely on zookeeper to ensure that the system availability cluster holds some meta information. 1. Producers

Producer publishes the message to the specified topic, and producer also decides which partition the message belongs to.

2. Consumers

Essentially Kafka only supports topic. Each consumer belongs to a consumer group; Conversely, there can be multiple consumer in each group. Messages sent to topic, will only be subscribed to one consumer consumption per group in this topic.
If all consumer have the same group, this is similar to the queue pattern, and the message will load evenly between consumers.
If all consumer have different group, this is "publish-subscribe"; The message will be broadcast to all consumers.

3. Topic

  A topic can be considered a kind of message, each topic will be divided into multiple partition (regions), each partition at the storage level is a append log file. Any messages published to this partition will be appended directly to the end of the log file, where each message's position in the file is called offset (offset), and the offset is a long number, which is uniquely marked with a single message. It uniquely marks a message. Kafka does not provide additional indexing mechanisms to store offset because it is almost impossible to read and write random messages in Kafka. Offset is stored in the zookeeper.

Kafka-topics.bat–create–zookeeper localhost:2181–replication-factor 1–partitions 3–topic page_visits

The Replication-factor node directory executes the following command (Replication-factor set to the number of Kafka nodes):
partitions– refers to the number of fragments
The above set topic command to create a 3 partition, placed only under one Replication node topic. 3. Partition 4. Distribution

About Kafka feature introduction can be more reference: http://www.cnblogs.com/likehua/p/3999538.html order Snapshot Log Analysis 1. Requirements

Want to get the order information from the gateway snapshot, the form of asynchronous sent to the backend, to compare the acquisition of the client after each order operation, the key elements of the change control. form the full log information within the order lifecycle. Display in the order details. 2. First edition design

Combined with Kafka, high availability, no repetition of message features, we use Kafka to do asynchronous buffering. The order snapshot is saved to the MongoDB first, then the quartz timing analysis is used. With the increasing amount of data, quartz can fail to analyze data in batches every time. Second, the need for business to display snapshot results in real time is more intense. 3. Second Edition design

When Kafka Cunsumer, do snapshot analysis directly. Because the order of the Kafka allows us to implement the snapshot comparison. When the message is fetched, it takes a snapshot of the previous step from the MongoDB and then makes a comparative analysis. The program worries about the speed of cunsumer consumption and whether broker will have a large backlog of messages. 

./kafka-consumer-groups.sh–zookeeper Localhost:2181/kafka–describe–group Group-1

By using the above command to specify your own groupings, the topic message record results are as follows:

GROUP	TOPIC	PID	OFFSET	LogSize	LAG
Consumer group	Topic ID	Partition ID	Number of currently consumed bars	Total number of articles	Number of strips not consumed

In the production environment, the application can be stable consumption, more stable than the first scheme, customer experience is more timely.