Some elk practices and elk practices

Some elk practices and elk practices
Some elk practices 0. Order

A recent task,
An elk is used for log analysis. logs are generated by 20 web servers at the same time, 108,555,691 records are generated in seven days, and about 2 million data records are collected in one minute, the peak is about two to four times of this.

Architecture Evolution: the architecture is not designed and evolved.

At the beginning, it was on a 2-Core 2 GB machine, including elasticsearch, logstash, and kibana.

Then a redis instance is added for broken. Redis encounters problems with kernel parameters, which may cause elasticsearch faults.

As the data volume increases, 2-Core 2 GB machines cannot meet the log pressure. Upgrade to 4-core and 4 GB.

Later, it was found that it was quite urgent because there were a lot of analysis items, such as the device of the geoip agent. Then you can purchase a 4-Core 4G instance and migrate elasticsearch and kibana to the new machine. In this way, the cpu and memory resources can be opened more than logstash.

The points of attention for each component are separated below

1. redis

It is very important to make a message queue.

2. logstash

Large cpu consumption, the cloud host must be configured with more than 4 cores, this can be opened more, the memory usage is generally about MB, mainly when we debug

stdout { codec => rubydebug }

But it must be closed during the official operation, which has a great impact on the performance.

3. elasticsearch

The most important thing is the core. You can use the rest api, but it is not very convenient. Fortunately, there is a plug-in! 2 recommended

Who knows who to use. These two plug-ins can be searched on the Internet slightly.

Elasticsearch automatically synchronizes data to the new node after a single node is used in the first step, so that each data is stored on two nodes, the elasticsarch status changes to green at this time.

But multiple nodes run, but the cluster runs a problem, it will greatly affect the logstash insertion efficiency. It was canceled later.

Install the head plug-in:

elasticsearch/bin/plugin -install mobz/elasticsearch-headopen http://localhost:9200/_plugin/head/

View the elasticsearch status:

curl 10.173.xx.xx:9200//_cluster/health?pretty{  "cluster_name" : "elasticsearch",  "status" : "yellow",  "timed_out" : false,  "number_of_nodes" : 2,  "number_of_data_nodes" : 1,  "active_primary_shards" : 53,  "active_shards" : 53,  "relocating_shards" : 0,  "initializing_shards" : 0,  "unassigned_shards" : 53,  "number_of_pending_tasks" : 0,  "number_of_in_flight_fetch" : 0}

Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.