Elasticsearch + Logstash + Kibana Configuration

Elasticsearch + Logstash + Kibana Configuration
Elasticsearch + Logstash + Kibana Configuration

There are many articles about the installation of Elasticsearch + Logstash + Kibana. I will not repeat them here, but I will only record some details here.

Precautions for installing AWS EC2

Remember to open the elasticsearch address on ports 9200,9300 and 5601. Do not write external IP addresses; otherwise, it will waste data and write internal ip addresses.

elasticsearch {host => "ip-10-160-94-102.ap-northeast-1.compute.internal"port => 9200cluster => "elasticsearch"protocol => http}

What pattern does grok have?

All lists are available here;

It is worth mentioning that QUOTEDSTRING and GREEDYDATA are good pattern when there are many texts;

Grok debugger is a powerful tool;

How can I lose unnecessary logs?

Grep filter is used on the Internet, but on the one hand, grep is not an official plug-in (but in logstash-contrib), and the new version does not seem to have been used. So I finally used only the if statement and grok.

The basic logic is

Each match adds a tag to determine whether a useful tag exists before the end of the tag.

 

if !(“tagA” in [tags] or “tagB” in [tags]) {
drop { }
}

Why can't the index value be sum/average? Why is my field not of the Number type but String type?

The grok document is very misleading. % {NUMBER: myfield} means to use the regular expression of a NUMBER to match, and the matched result grok is indexed as a string by default.

To specify the index type, you must enter % {NUMBER: myfield: int} And: float.

If the index of a field is already string, you can use the Mapping API of elasticsearch to change it to int, but I have not succeeded. My final success was to delete all the previous indexes and redo them.

$ curl -XDELETE ‘http://localhost:9200/twitter/’

How to Create visualization?

I don't understand a lot of things either. If you have the energy, you can read the documents written by the great gods. Let's briefly describe some basic principles and tips:

"Split" is equivalent to setting the size to 0 after the group by "search" is created in "discover" terms. If no limit terms is selected, you can use include or exclude to limit this field, equivalent to where

Finally, I posted a piece of mutual encouragement for three days: