How do I configure an index template for Logstash+elasticsearch?

When we use Logstash to collect logs, we usually use the dynamic Index template that comes with logstash, although we can push our log data to the Elasticsearch index cluster without any custom action, but when we query, we find that The default index template often puts us in a field that does not need a word breaker, so that our more important aggregated statistics are inaccurate:

For example, if there are 10 machines in need of monitoring, their machine name is as follows:

Java code

search-0-170

search-1-171

search-2-172

search-3-173

search-4-174

search-5-175

search-6-176

search-7-177

search-8-178

search-9-179

If you are using the default template for Logstash, it will press-to split the machine name, so that you want to count the collection log on that machine at most there is a problem, so this time, we need to customize some of the index template:

When integrating Logstash with Elasticsearch, there are a total of several ways to use templates:

(1) using the default own index template, most of the fields will be participle, suitable for development and time to quickly verify the use of
(2) in the Logstash collection end custom configuration template, because scattered on the collection machine, maintenance is more troublesome
(3) in the ELASTICSEARC server custom configuration template, the Elasticsearch is responsible for loading the template, can be changed dynamically, global effective, maintenance is relatively easy

The above several ways:

Use the first, the simplest, without any configuration
Using the second, log collection for small-scale clusters, you need to specify a template JSON path on the machine in the output plug-in for Logstash, such as templates = "/tmp/logstash.json"
Using the third type, which is suitable for large-scale cluster log collection, how to configure, the main configuration Logstash output plug-in two parameters:

Java code

Manage_template = false//Close Logstash Auto Manage templates feature

Template_name = Name of "Crawl"//Map template

If used, the third needs to configure the template JSON under the Config/templates path in the Elasticsearch cluster, and the index template can be divided into two types in Elasticsearch:

(i): static template

Suitable for fixed index field data scene, once the configuration is complete, you can not add extra fields to the inside, otherwise it will be an error

Advantages: Scheam Known, business scenario is clear, not easy to appear due to the field random mapping resulting in meta-data burst ES memory, resulting in ES cluster all downtime
Cons: Configuration is a bit cumbersome in the case of multiple fields

A static index template configuration example is as follows:

JSON code

{

"Crawl": {

"Template": "crawl-*",

"Settings": {

"Index.number_of_shards": 3,

"Number_of_replicas": 0

},

"Mappings": {

"Logs": {

"Properties": {

"@timestamp": {

"Type": "Date",

"Format": "Dateoptionaltime",

"Doc_values": True

},

"@version": {

' Type ': ' String ',

"Index": "Not_analyzed",

"Doc_values": True

},

"CID": {

' Type ': ' String ',

"Index": "Not_analyzed"

},

"Crow": {

' Type ': ' String ',

"Index": "Not_analyzed"

},

"Erow": {

' Type ': ' String ',

"Index": "Not_analyzed"

},

"Host": {

' Type ': ' String ',

"Index": "Not_analyzed"

},

"Httpcode": {

' Type ': ' String ',

"Index": "Not_analyzed"

},

"Message": {

' Type ': ' String '

},

"Path": {

' Type ': ' String '