Logstash+kafka for real-time Log collection _ non-relational database

Using spring to consolidate Kafka only supports kafka-2.1.0_0.9.0.0 and above versions

Kafka Configuration

View Topic

bin/kafka-topics.sh--list--zookeeper localhost:2181


Start a producer


bin/kafka-console-producer.sh--broker-list localhost:9092--topic test


Open a consumer (2183)


bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic test--from-beginning


Create a Theme
bin/kafka-topics.sh--create--zookeeper 10.92.1.177:2183--replication-factor 1--partitions 1--topic test

Logstash Boot log capture
Bin/logstash-f logstash-simple.conf

Important filter condition of filter not in drop{}--Delete

Logstash configuration file:

 input{Kafka {codec => "plain" group_id => "Logstash1" Auto_offset_reset => "SM Allest "Reset_beginning => true topic_id =>" Hello "#white_list => [" Hello "] #bla Ck_list => nil zk_connect => "192.168.0.5:2181" # Zookeeper address}}filter {  If "xml" not in [Messa  GE] {           drop {}  }   If "success" not in [message]{          Drop {}  }} output{stdout{codec => rubydebug}}