Use logstash to collect php-fpmslowlog

Use logstash to collect php-fpmslowlog. Currently, the php-fpm service is deployed in docker. the php-fpm log and php error log can be sent through the syslog protocol, however, the slow log of php-fpm cannot be configured as a syslog protocol and can only be output to files, because an slow log consists of multiple lines.

To collect slow logs, you can collect them using tools such as logstash and flume. This article uses logstash to collect slow logs and write the collected logs to kafka for subsequent processing. Logstash input adopts the file reading method, which is similar to the principle of tail-f. To merge multiline logs as one row, multiline logs in the filter are used. The logstash configuration is as follows:

Input {file {path => ["/var/log/php-fpm/fpm-slow.log"]} filter {multiline {pattern => "^ $" negate => true what => "previous" }}output {stdout {codec => rubydebug} kafka {codec => plain {format => "tag |%{ host }%{ message}"} topic_id => "fpm-slowlog" bootstrap_servers => "kafka1.hostname: 8082, kafka2.hostname: 8082 "}}