Logstash Log collection display and email alerts

Sometimes we need to analyze some server logs and alarm the wrong logs, where we use Logstash to collect these logs and send error log data using our own developed mail delivery system.

For example we have several files that need to be monitored (BI logs)

We can collect these file logs by configuring Logstash

input{ file{ Path=> "/diskb/bidir/smartbi_prd_*/apache-tomcat-5.5.25_prd_*/logs/catalina.out" Start_position=> "Beginning" Sincedb_path=> "/diskb/logstashlog/log" Codec = plain { CharSet = "GBK" } } } Filter { Multiline { Pattern = "^\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}" Negate = True what = "Previous" } mutate{ Add_field = { "LogMessage" = "%{[message]}" } split = ["Message", "" "] Add_field = { "Logdate" = "%{[message][0]}" "LogTime" = "%{[message][1]}" "Logstate" = "%{[message][2]}" } Remove_field = ["Message"] } If [logdate]!~/\d{2}-\d{2}/{ drop{} } UrlDecode { All_fields = True } } output{ #对错误的日志写入到文件中, attachments used for e-mail delivery #其实在这里可以直接使用logstash自带的邮件发送系统, only the department requires that the frequency of the mail be sent #高于一分钟, it can only be sent by external timing If [logstate] =~/error/{ File { Path = "/diskb/bi_error_log/bi_error.log" } } elasticsearch{ hosts = ["10.130.2.53:9200", "10.130.2.46:9200", "10.130.2.54:9200"] flush_size=>50000 Workers = 5 Index=> "Logstash-bi-tomcat-log" } }

By starting this conf file, you can import all the data into ES, can be displayed by Kibana, the specific display will not repeat, and at the same time the error log is imported into a text for the mail sending system to use. This is over.

Attached: Send mail script

#!/bin/sh #sendmail Error log to someone   #发送的附件路径 attachement= "/diskb/bi_error_log/*.log"   If [!-F $attachement];then Echo "File is not exist" Exit 1 Fi   #收件人 Maillist= "[email protected]"    cat >/etc/nail.rc<<eof Set [email protected] Set smtp=60.28.250.158 Set [email protected] set smtp-auth-password=****** Set Smtp-auth=login EOF   #echo Mail Content |/usr/local/mailx-12.4/mailx-v-S "Message title" [-A "attachment path"] [-C "Bcc Mail"] recipient Echo "Hello, please receive the error log for BI from HEXUN.BDC." |/usr/local/mailx-12.4/mailx-v-S "[The system send S] "-a $attachement $maillist     #发送成功, delete files rm-fr $attachement

Logstash Log collection display and email alerts