When we use Logstash to collect logs, we usually use the dynamic Index template that comes with logstash, although we can push our log data to the Elasticsearch index cluster without any custom action, but when we query, we find that The default index template often puts us in a field that does not need a word breaker, so that our more important aggregated statistics are inaccurate:
For example, if there are 10 machines in need of monitoring, their machine name is as follows:
Java code
search-0-170
search-1-171
search-2-172
search-3-173
search-4-174
search-5-175
search-6-176
search-7-177
search-8-178
search-9-179
If you are using the default template for Logstash, it will press-to split the machine name, so that you want to count the collection log on that machine at most there is a problem, so this time, we need to customize some of the index template:
When integrating Logstash with Elasticsearch, there are a total of several ways to use templates:
(1) using the default own index template, most of the fields will be participle, suitable for development and time to quickly verify the use of
(2) in the Logstash collection end custom configuration template, because scattered on the collection machine, maintenance is more troublesome
(3) in the ELASTICSEARC server custom configuration template, the Elasticsearch is responsible for loading the template, can be changed dynamically, global effective, maintenance is relatively easy
The above several ways:
Use the first, the simplest, without any configuration
Using the second, log collection for small-scale clusters, you need to specify a template JSON path on the machine in the output plug-in for Logstash, such as templates = "/tmp/logstash.json"
Using the third type, which is suitable for large-scale cluster log collection, how to configure, the main configuration Logstash output plug-in two parameters:
Java code
Manage_template = false//Close Logstash Auto Manage templates feature
Template_name = Name of "Crawl"//Map template
If used, the third needs to configure the template JSON under the Config/templates path in the Elasticsearch cluster, and the index template can be divided into two types in Elasticsearch:
(i): static template
Suitable for fixed index field data scene, once the configuration is complete, you can not add extra fields to the inside, otherwise it will be an error
Advantages: Scheam Known, business scenario is clear, not easy to appear due to the field random mapping resulting in meta-data burst ES memory, resulting in ES cluster all downtime
Cons: Configuration is a bit cumbersome in the case of multiple fields
A static index template configuration example is as follows:
JSON code
{
"Crawl": {
"Template": "crawl-*",
"Settings": {
"Index.number_of_shards": 3,
"Number_of_replicas": 0
},
"Mappings": {
"Logs": {
"Properties": {
"@timestamp": {
"Type": "Date",
"Format": "Dateoptionaltime",
"Doc_values": True
},
"@version": {
' Type ': ' String ',
"Index": "Not_analyzed",
"Doc_values": True
},
"CID": {
' Type ': ' String ',
"Index": "Not_analyzed"
},
"Crow": {
' Type ': ' String ',
"Index": "Not_analyzed"
},
"Erow": {
' Type ': ' String ',
"Index": "Not_analyzed"
},
"Host": {
' Type ': ' String ',
"Index": "Not_analyzed"
},
"Httpcode": {
' Type ': ' String ',
"Index": "Not_analyzed"
},
"Message": {
' Type ': ' String '
},
"Path": {
' Type ': ' String '