ELK Kafka JSON to ELK

Logstash Configuration

??

Input {

Kafka {

Zk_connect = "127.0.0.1:2181"

TOPIC_ID = "Cluster"

codec = Plain

Reset_beginning = False

Consumer_threads = 5

Decorate_events = True

}

}

??

??

Output {

If [type]== "Cluster3" or [type]== "Cluster2" or [type]== "Clusterjson"

{

Elasticsearch {

hosts = ["localhost:9200"]

index = "test-kafka-%{type}-%{+yyyy-mm}"

}

}

??

stdout {codec = Rubydebug}

}

??

Server.properties Main Content

Broker.id=0

??

############################# Socket Server Settings #############################

??

listeners=plaintext://:9092

??

# The port the socket server listens on

port=9092

??

??

Server-1.properties Main Content

??

Broker.id=1

??

############################# Socket Server Settings #############################

??

listeners=plaintext://:9093

??

# The port the socket server listens on

port=9093

??

??

Start Kafka,server.properties and Server-1.properties

E:\soft\elk\kafka_2.11-0.9.0.1>bin\windows\kafka-server-start.bat config\custom\server-1.properties

??

Create a topic and view the status

??

??

Find port occupancy and delete a node

??

??

View topic Status

??

??

??

The producer enters the JSON data as follows

??

{"Name": "BULK 7", "Data": 7, "Createtime": "2016-04-05t10:16:22z", "type": "Cluster3"}

??

Kibana 4 shows the following

??

??

??

??

??

??

??

ELK Kafka JSON to ELK