Code dry |logstash Detailed--filter module

Article from Aliyun-yun-Habitat community, the original click here.

The second component of the Logstash three components is also the most complex, logstash component of the entire tool, and, of course, the most useful component.
1, Grok plug-in Grok plug-in has a very powerful function, he can match all the data, but his performance and the loss of resources also let people criticized.

filter{

    grok{
        #只说一个match属性, his role is to extract the time from the Message field, and assign a value to another field logdate.
        #首先要说明的是, all text data is in the Logstash message field, and the data we want to manipulate in the filter is the message.
        #第二点需要明白的是grok插件是一个十分耗费资源的插件, that's why I'm only going to explain the reason for a timestamp_iso8601 regular expression.
        #第三点需要明白的是, Grok has a lot of preloaded regular expressions, there's no way to get it all done, maybe you can find the expression you need from this great God article
        #http://blog.csdn.net/liukuan73/ article/details/52318243
        #但是, I still do not recommend it, because he can be replaced by other plug-ins, of course, for the property of time, Grok is very convenient.
        match => [' message ', '%{timestamp_iso8601:logdate} '}
}

2, mutate plug-in mutate plug-ins are used to handle the format of the data, you can choose to process your time format, or you want to make a string into a numeric type (of course, you need legal), the same you can return to do. The types of conversions that you can set include: "Integer", "float", and "string".

Filter {
    Mutate {
        #接收一个数组, in the form of Value,type
        #需要注意的是, your data in the transformation of the legal, you always can not convert an ' ABC ' string to 123.
        convert => [
                    #把request_time的值装换为浮点型
                    "Request_time", "float",
                    #costTime的值转换为整型
                    "Costtime", ' Integer '
    }
}

3, the Ruby plug-in official introduction to the Ruby plug-in is-omnipotent. Ruby Plug-ins can use any ruby syntax, whether it's a logical judgment, conditional statements, loop statements, or manipulation of strings, which are extremely handy for the event object.

Filter {ruby {#ruby插件有两个属性, an init also has a code #init属性是用来初始化字段的, you can initialize a field here, no matter what type, this field is just ruby{}
        The scope takes effect. #这里我初始化了一个名为field的hash字段.
        Can be used in the coed attribute below.
        Init => [field={}] #code属性使用两个冒号进行标识, all of your Ruby syntax can be done inside.
        #下面我对一段数据进行处理. #首先, I need to get the value in the Message field and split the value according to "|".
        This split out is an array (Ruby's character creation).
        #第二步, I need to loop the array to determine whether the value is the data I need (ruby conditional syntax, looping structure) #第三步, I need the fields I need to add to the Evevt object. #第四步, select a value for MD5 encryption #什么是event对象.
        Event is the Logstash object, you can in the Ruby plug-in code attributes inside the operation of him, you can add property fields, you can delete, you can modify, the same can be done resin operation.
        #进行MD5加密的时候, the corresponding package needs to be introduced.
        #最后把冗余的message字段去除. Code => "Array=event.
            Get (' message '). Split (' | ')
                Array.each do |value| If Value.include? ' Md5_value ' then require ' digest/md5 ' md5=digest::md5 . Hexdigest (value) event.set (' MD5 ', MD5) End if Value.include? ' DefauLt_value ' then Event.set (' value ', value) end

 remove_field=> "Message"}}

4, the date plug-in here needs to be combined with the front of the Grok plug-in value logdate with the use of (of course, you may not use Grok to do).

filter{
    date{
        #还记得grok插件剥离出来的字段logdate吗. That's what you use here. You can format it as you want it to be, as to what it looks like. You'll have to take a look at it yourself.
        #为什什么要格式化.
        #对于老数据来说这非常重要, should you need to modify the value of the @timestamp field, if you do not modify, you save the ES time is the system but the previous time (+0 time zone)
        #单你格式化以后, You can specify the target attribute to @timestamp, so that your data will be accurate, which is extremely important for your future diagrams.
        #最后, logdate This field has no value anymore, so we can do it. This field is removed from the event object.
        match=>["Logdate", "Dd/mmm/yyyy:hh:mm:ss Z"]
        target=> "@timestamp"
        remove_field => ' logdate '
        #还需要强调的是, @timestamp the value of the field, you can not arbitrarily modify, it is best to follow your data at a certain point in time to use,
        #如果是日志, use Grok to pull out the time, if it is a database, specify a field value to format, For example: "Timeat", "%{timestamp_iso8601:logdate}"
        #timeat就是我的数据库的一个关于时间的字段.
        #如果没有这个字段的话, don't try to modify it.

    }
}

5, JSON Plug-ins, this plug-in is also extremely useful for a plug-in, now our log information

>>> Expand Full text