Data acquisition of Kafka and Logstash

Data acquisition of Kafka and Logstash

Based on Logstash run-through Kafka still need to pay attention to a lot of things, the most important thing is to understand the principle of Kafka.

Logstash Working principle

Since Kafka uses decoupled design ideas, it is not the original publication subscription, the producer is responsible for generating the message and pushing it directly to the consumer. Instead, the persistence layer is added to the--broker, where the producer stores the data in the broker and the consumer takes the data from the broker. This brings a few benefits:

• 1 The load of the producer is decoupled from the consumer's load
• 2 consumers fetch data according to their abilities
• 3 Consumers can customize the amount of consumption

In addition, because the broker adopts the idea of topic topic--> partitioning, the order of a partition is guaranteed to be orderly, but the data between partitions is not guaranteed to be orderly. In this way, the consumer can--offset the location of the custom read in the partition unit.

Kafka uses zookeeper as the management to record producer to broker information and consumer's correspondence with partition in broker. As a result, producers can transmit data directly to broker,broker through zookeeper for leader-->followers election management The consumer saves the read location offset as well as the partition partition information of the topic read through zookeeper.

Because of the architecture design above, the producer is connected with the broker; the consumer is connected with the zookeeper. With this correspondence, it is easy to deploy the Logstash-->kafka-->logstash solution.

Next, follow the steps below to achieve Logstash and Kafka docking.

Start Kafka

Start Zookeeper:

$zookeeper/bin/zkServer.sh start

Start Kafka:

$kafka/bin/kafka-server-start.sh $kafka/config/server.properties &

Create a Theme

To create a theme:

127.0.0.1:2181 --create --topic hello --replication-factor 1 --partitions 1

To view topics:

--zookeeper 127.0.0.1:2181 --describe

Test environment

Execute producer Script:

$kafka/bin/kafka-console-producer.sh --broker-list 10.0.67.101:9092 --topic hello

Execute the consumer script to see if the write:

$kafka/bin/kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --from-beginning --topic hello

Input test

input{    stdin{}}output{    kafka{        "hello"        bootstrap_servers => "192.168.0.4:9092" # kafka的地址        batch_size => 5    }    stdout{        codec => rubydebug    }}

Read Test

Logstash configuration file:

input{    kafka {        "plain"        group_id => "logstash1"        auto_offset_reset => "smallest"        reset_beginning => true        topic_id => "hello" #white_list => ["hello"] #black_list => nil zk_connect => "192.168.0.5:2181" # zookeeper的地址 }}output{ stdout{ codec => rubydebug }}

Category: Kafka, Logstash

Data acquisition of Kafka and Logstash