Structure of log analysis

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/59/DF/wKiom1TtkATgkILyAAJp-BJblac839.jpg "title=" Log Analysis _ 1.png "alt=" Wkiom1ttkatgkilyaajp-bjblac839.jpg "/>

running various applications on the production server, these programs are scripted by a log check (logcheck.sh) periodic scanning. Use crontab to control the frequency of scanning. After each scan, the script will record the last line of the log for this scan, so that the next scan can be started directly from the newly generated journal to avoid duplicate scans. The scan-found exception is added to the log file under a unified directory, which is followed by the server and is used to name it, ensuring that each application on each machine has a separate log output file. If you have multiple identical apps on a single server, you need to differentiate each app.

another script (maillog.py) the script will look at this directory every half hour or a certain time, if found to have when a new file is generated, the file is notified to the OPS group via email and SMS. If the application generates a small amount of log (<300m), you can increase the frequency of the run (logcheck.sh) appropriately, which allows for a low latency.

the log files generated by the production server need to be synchronized to the log server. Use rsync technology to synchronize log files on the server. If the performance of the log server is allowed and latency is not the primary consideration, the log check script mentioned above can also be placed on this server to run. a set of Logstash + Elasticsearch is built on the log server . Logs on the log server are read into the Elasticsearch database and then provided for development and operational use through the Logstash interface.

a better solution is to Logstash combined RabbitMQ read into the log message flow, which enables approximate real-time processing of logs. Then analyze the flow of these messages, and once you find an exception, email and SMS alerts.

This article from the "Automated Operation and Maintenance" blog, reproduced please contact the author!

Structure of log analysis