Tutorial: Use rsyslog to push logs to kafka and elasticsearch

Tutorial: Use rsyslog to push logs to kafka and elasticsearch

This article introduces a simple method for pushing logs to kafka and elasticsearch using rsyslog, installing and using the rsyslog omkafka plug-in, and installing and using the rsyslog omelasticsearch plug-in.

Kafka is an open-source distributed message system. The project homepage is kafka.apache.org.
Elasticsearch is an open-source distributed search engine. Project homepage: elastic. co

Rsyslog uses omkafka to push logs to kafka and omelasticsearch to push logs to elasticsearch. By default, the compilation options of these two plug-ins are disabled and are not compiled into rsyslog. The following describes the installation method:

## add rsyslog repoWORK_DIR=$(pwd)cd /etc/yum.repos.dwget http://rpms.adiscon.com/v8-stable/rsyslog.repo -O rsyslog.repocd $WORK_DIRmkdir rsyslog-installcd rsyslog-install# check rsyslog version# rsyslog supports kafka from v8.7.0old_rsyslog_ver=$(rsyslogd -version |head -n 1 | awk '{print $2}')## install rsyslog dependency: libestryum install -y libestr-devel## install rsyslog dependency: libeeyum install -y libee-devel## install rsyslog dependency: json-cyum install -y json-c-devel## install rsyslog denpendency: uuidyum install -y libuuid-devel## install rsyslog denpendency: liblogging-stdlogyum install -y liblogging-devel## install rsyslog denpendency: rst2manyum install -y python-docutils## install libcurl for omelasticsearchyum install -y libcurl-devel## install librdkafka for omkafkawget https://github.com/edenhill/librdkafka/archive/0.8.5.tar.gz -O librdkafka-0.8.5.tar.gztar zxvf librdkafka-0.8.5.tar.gzcd librdkafka-0.8.5./configuremakemake installcd ..## install rsyslogwget http://www.rsyslog.com/files/download/rsyslog/rsyslog-8.8.0.tar.gz -O rsyslog-8.8.0.tar.gztar zxvf rsyslog-8.8.0.tar.gzexport PKG_CONFIG_PATH=/usr/lib64/pkgconfig:/lib64/pkgconfig/old_executable_path=$(which rsyslogd)executable_dir=$(dirname "$old_executable_path")cd rsyslog-8.8.0./configure --sbindir=$executable_dir --libdir=/usr/lib64 --enable-omkafka --enable-elasticsearchmakemake install## show installation result:new_rsyslog_ver=$(rsyslogd -version |head -n 1 | awk '{print $2}')echo "Old rsyslogd version: "$old_rsyslog_verecho "New rsyslogd version: "$new_rsyslog_verecho "Executable: " $(which rsyslogd)

I have hosted the relevant code on Github:
Https://github.com/garyelephant/rsyslog-scripts

For more information about the omkafka plug-in, see:
Http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html

For more information about the omelasticsearch plug-in, see:
Http://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html

Configuration example:

# /etc/rsyslog.conf# load required module# `imuxsock` provides support for local system logging (e.g. via logger command)module(load="imuxsock") module(load="omkafka")module(load="omelasticsearch")# push to kafkaaction(type="omkafka" topic="your_topic" broker="your_kafka_broker_host_or_ip")# or you can push to elasticsearchaction(type="omelasticsearch" server="your_elasticsearch_host_or_ip" searchIndex="your_elasticsearch_index" searchType="your_elasticsearch_index_type" )

Start rsyslog

rsyslogd -n

Use logger to write data to rsyslog on another terminal

$ logger 'hello world'

References: http://www.rsyslog.com/doc/master/installation/install_from_source.html http://bigbo.github.io/pages/2015/01/21/syslog_kafka/ http://blog.oldzee.com/?tag=rsyslog http://www.rsyslog.com/newbie-guide-to-rsyslog/ http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html