ELK Beats Platform Introduction (11th)

Beats is a proxy that sends different types of data to Elasticsearch. Beats can send data directly to Elasticsearch, or you can send the data elasticsearch through Logstash.

Beats has three typical examples: Filebeat, Topbeat, Packetbeat. Filebeat is used to collect logs, topbeat is used to collect the system basic settings data such as CPU, memory, each process statistics, packetbeat is a network packet analysis tool, statistical collection of network information. These three are officially provided. Follow-up will introduce these three beat slowly.

The goal of Elk is to build a platform that can easily create new beats. For this directory, Libbeat was developed and the Go Library contains all the beats public parts to handle tasks such as bulk insertion into elasticsearch, secure send events to Logstash,logstash and Elasticsearch Multi-node load balancing, The asynchronous or synchronous send event pattern. The Libbeat platform also provides a detection mechanism that automatically reduces the transmit rate when the downstream server is under high load or network congestion.

The architecture diagram is as follows:

In short, libbeat can safely and reliably send all events to Logstash and Elasticsearch. Not only that, it also takes into account other things such as configuration, CLI tags, and logs. So when you create a new beat, you just need to focus on capturing the data you want. Other parts of the analysis platform were handed over to Libbeat, Logstash, Elasticsearch and Kibana. such as the community provides Dockerbeat, Pingbeat, Uwsgibeat and Nginx beat.

Topbeat

Collects system load, memory, hard disks, and the situation of each process.

The Topbeat is cross-platform. Periodically send the indicator to Elasticsearch. You can create custom dashboards through Kibana such as system load, server overview, memory or CPU usage, top process, CPU or memory ratio per process, disk usage, and so on.

Packetbeat

Imagine if you want to monitor and solve a complex distributed system, each component uses a different programming language, Web framework, and database technology. Packetbeat is distributed, in real time, sniffing the requests and responses of each transaction and inserting the relevant data into the Elasticsearch. Packetbeat passively sniffing network traffic and therefore does not interfere with the application.

Packetbeat is open source and anyone can add support for new protocols. The community added support for MongoDB, UDP and TCP-based DNS support, and so on.

Of course, you can also add new support. Look forward to ING

Filebeat

Filebeat is a substitute for Logstash forwarder. This blog before the article is introduced, "Filebeat log collector Logstash-forwarder alternatives."

The log is logstash further processed and sent to the Elasticsearch. There is enough intelligence to handle log rotation, file renaming, and downstream servers being unavailable, so don't worry about losing data.

It is strongly recommended that the Logstash forwarder be migrated to Filebeat and Logtash-forwarder migrated to Filebeat.

The following is a brief introduction to these three beats, which are later provided by the community beat. Please pay attention.

ELK Beats Platform Introduction (11th)