From Logstash, output, elasticsearch dynamic template

Logstash Index Mappings

"Mappings": {"_default_": {"dynamic_templates": [{"String_fields": {
                        "Mapping": {"index": "Analyzed", "omit_norms": true,
                                "Type": "String", "fields": {"raw": {
                                "Index": "Not_analyzed", "Ignore_above": 256, "Type": "String"}}, "M Atch ":" * "," Match_mapping_type ":" string "}]," _all ": {"Enabled": true}, "Properties": {"@version": {"type": "String
                "," index ":" Not_analyzed "}," GeoIP ": {" dynamic ":" true ", "Properties":{"Location": {"type": "Geo_point"}} }}}, "Hkt_log": {"dynamic_templates": [{"String_fields" : {"Mapping": {"index": "Analyzed", "omit_norms": TR
                                UE, "type": "String", "fields": {"raw": {
                                "Index": "Not_analyzed", "Ignore_above": 256,
                    "Type": "String"}},
        "Match": "*", "Match_mapping_type": "String"}}],
                "_all": {"Enabled": true}, "Properties": {"@timestamp": {
        "Type": "Date",        "Format": "Dateoptionaltime"}, "@version": {"type": "String", "Index": "Not_analyzed"}, "GeoIP": {"dynamic": "true", "PR
                Operties ": {" location ": {" type ":" Geo_point "}
                    }}, "host": {"type": "string", "norms": {
                        "Enabled": false}, "fields": {"raw": { 
                    "Type": "string", "index": "Not_analyzed", "Ignore_above": 256
                }}}, "message": {"type": "String",
                    "Norms": {"Enabled": false}, "fields": {
             "Raw": {           "Type": "string", "index": "Not_analyzed", "Ignore_above": 256
                }}}, "path": {"type": "String", "Norms": {"Enabled": false}, "fields": {"R
                        Aw ": {" type ":" string "," index ":" Not_analyzed ", "Ignore_above": "{"}}, "type": {"type": "
                    String "," norms ": {" Enabled ": false}," fields ": {
                        "Raw": {"type": "string", "index": "Not_analyzed", "Ignore_above": 256}}}}

_ default _ Map

When the output plug-in for Logstash is Elasticsearch, Logstash will automatically create a "default" map for the current index data in Es, followed by all newly created type Mapping will inherit this default mapping.
This default mapping, the _all field defaults to Disabled, where it is enabled
Add @version field by default
GeoIP field Dynamic True indicates that a new field can be added dynamically under this field

The following type Hkt_log inherits the mapping content of default. dynamic_templates

With Dynamic_templates, you can have complete control over the dynamic mapping rules for new fields. You can use a different mapping rule depending on the field name or type.

Each template has a name that can be used to describe what the template does. At the same time it has a mapping used to specify specific mapping information, and at least one parameter (such as match) is used to stipulate what fields need to be used for the template.

The matching of templates is sequential-the first matching template is used.

The above Logstash template defines a statistical setup template for all string fields:

{"
    String_fields": {"mapping": {"
            index": "Analyzed",
            "omit_norms": True,
            "type": "String",
            "Fields": {
                "raw": {"
                    index": "Not_analyzed",
                    "Ignore_above": "$",
                    "type": "String"
                }
            }
        },
        "Match": "*",
        "Match_mapping_type": "String"
    }
}

String_fields is the template name, and all string fields are defined as uniform mappings. Match_mapping_type allows you to use templates only for specific types of fields, such as the standard dynamic mapping rules, such as String,long. The match parameter matches only the field name.

The Path_match parameter is also used to match the full path of the fields in the object, such as Address.*.name can match the following fields:

{
    "address":
        "City":
            "name": "New York"}}
}

Unmatch and Path_unmatch modes can be used to exclude certain fields, and fields that are not excluded will be matched.

Fields is a multi-field property, and if there is a field called message, then there is a sub-segment Message.raw, which holds the original contents of the Message field and is not parsed by the parser. "Index": "not_analyzed" does not parse field contents "Ignore_above": 256 longest processing 256 characters "type": "string" type hkt_log type mapping

We can see that the mapping of the Hkt_log type completely inherits the mapping of the default, while adding a specific field dynamically, but you can see that all the string fields follow the definition of the String_fields template, such as the message field:

"Message": {
    "type": "string",
    "norms": {
        "Enabled": false
    },
    "Fields": {
        "raw": {
            "type ":" string ",
            " index ":" Not_analyzed ",
            " Ignore_above ": [
        ]
    }
}

The message will have a sub-segment raw that holds the original content that was not parsed by the parser.

That's a lot more. What are the benefits of such a field?
Let's first add a document to the Hkt_log:

Put LOGSTASH-2015.10.01/HKT_LOG/1
{
    "message": "Jfy/zhyy?123",
    "@version": "1",
    "@timestamp": "2015-11-12t03:14:41.435z",
    "type": "Hostapd1_log", "
    Host": "server114",
    "path": "/root/hostapd/ Hostapd1.log "
}

The following query contains "?" in the Message field. The Documentation:

{"
  query": {"regexp": {"
      message": {"value": ". *\\?". *"}
    }
  }
}

Cannot find the result, change to:

{"
  query": {"regexp": {"
      Message.raw": {"value": ". *\\?". *"}
    }
  }
}

can query to.

This is because the message field content "Jfy/zhyy?123" is parsed by the parser and becomes three words: jfy,zhyy,123, where the "/" and "?" The parser is ignored.

POST logstash-2015.10.01/_analyze
jfy/zhyy?123

{"
    tokens": [
        {
            "token": "JFY",
            "Start_ Offset ": 0,
            " End_offset ": 3,
            " type ":" <ALPHANUM> ",
            " position ": 1
        },
        {
            " token ":" Zhyy ",
            " Start_offset ": 4,
            " End_offset ": 8,
            " type ":" <ALPHANUM> ",
            " position ": 2
        },
        {
            "token": "123",
            "Start_offset": 9,
            "End_offset": "
            Type": "<NUM>",
            " Position ": 3
        }
    ]
}

The Message.raw is stored as-is, so it can be queried with wildcard characters.