FLUENTD Push MARIADB Audit log

Description

The MARIADB audit log is the audit log for MARIADB

The goal is to split the log into tab-delimited fields

Attach FLUENTD configuration file directly

<system>Log_level Error</system><source>@typeTailPath/data/logs/mariadb/server_audit.log tag Mysql_audit pos_file/data/logs/mariadb/Fluentd.pos<parse>@type Multiline Format_firstline/^\d{8}/FORMAT1/^ (? <dt>\d{8} \d{2}:\d{2}:\d{2}), (?<hostname>[^,]+), (<user>[^,]+), (? <ip>[^,]+), (? <connid>[^,]+), (? <queryid>[^,]+), (? <action >[^,]+), (? <db>[^,]+), (? <message>.*), (? <retcode>\d+) $/</parse></source>< Filter mysql_audit>@typegrep<regexp>Key action Pattern QUERY</regexp> <exclude>Key user Pattern Lagou_status</exclude> <exclude>Key db pattern Information_schema</exclude></filter><filter mysql_audit>@type Record_transformer Enable_ruby<record>message ${record["message"].gsub (/\s/,' ')} message ${record["message"].gsub (/\s+/,' ')}  </record></filter><match mysql_audit># @type stdout @typefilePath"/data/logs/mariadb/#{env[' HOSTNAME '}"Time_slice_format%y%m%d%H time_slice_wait 5m Add_path_suffixfalseAppendtrueCompressgzip<format>@type csv fields DT,hostname, User,ip,action,db,message,retcode delimiter'    '</format></match>

FLUENTD memory footprint significantly lower than logstash

Analyze the same log logstash occupy 700M, Fluentd occupy 35M

But CPU occupied quite, for the log volume of large machine CPU to 100%

It seems that regular filtering of logs is a drain on the CPU.

FLUENTD Push MARIADB Audit log