Use the latest open-source software to build a log management system

We already know that OSSIM is one of the few open-source SIEM/security management platforms, and there is no integrated log management (LM) system yet.

However, if you want to, you can DIY a log management system and use the latest technologies.

First, you need to use logstash to collect logs. It has a long history, but is very trendy. It supports collecting logs in N ways and outputting logs in N ways. This is a great log collector.

Of course, logstash also supports AMQP. Therefore, if you are planning to build a distributed collector, you can consider combining AMQP with logstash.

Then, another core of LM is historical analysis. You need to use elasticsearch, which is also a cutting-edge product of Lucene. Logstash can provide logs to elasticsearch, store logs in history and index the full text, and provide a web ui for full-text retrieval.

Well, as mentioned above, there is now basically a trendy LM architecture.

If you still don't think it is enough, you just want to do some real-time analysis. Okay, you may need graylog2. Its Background storage can use MongoDB, and its log filtering capability is also good.

Finally, can all these integrated open-source LM replace commercial products? There are at least two issues to consider-performance and stability! Of course, maintenance and optimization are also difficult.