Tutorial: Use rsyslog to push logs to kafka, elasticsearch, and rsyslogkafka

Source: Internet
Author: User
Tags rsyslog

Tutorial: Use rsyslog to push logs to kafka, elasticsearch, and rsyslogkafka

This article introduces a simple method for pushing logs to kafka and elasticsearch using rsyslog, installing and using the rsyslog omkafka plug-in, and installing and using the rsyslog omelasticsearch plug-in.

Kafka is an open-source distributed message system. The project homepage is kafka.apache.org.
Elasticsearch is an open-source distributed search engine. Project homepage: elastic. co

Rsyslog uses omkafka to push logs to kafka and omelasticsearch to push logs to elasticsearch. By default, the compilation options of these two plug-ins are disabled and are not compiled into rsyslog. The following describes the installation method:

## add rsyslog repoWORK_DIR=$(pwd)cd /etc/yum.repos.dwget http://rpms.adiscon.com/v8-stable/rsyslog.repo -O rsyslog.repocd $WORK_DIRmkdir rsyslog-installcd rsyslog-install# check rsyslog version# rsyslog supports kafka from v8.7.0old_rsyslog_ver=$(rsyslogd -version |head -n 1 | awk '{print $2}')## install rsyslog dependency: libestryum install -y libestr-devel## install rsyslog dependency: libeeyum install -y libee-devel## install rsyslog dependency: json-cyum install -y json-c-devel## install rsyslog denpendency: uuidyum install -y libuuid-devel## install rsyslog denpendency: liblogging-stdlogyum install -y liblogging-devel## install rsyslog denpendency: rst2manyum install -y python-docutils## install libcurl for omelasticsearchyum install -y libcurl-devel## install librdkafka for omkafkawget https://github.com/edenhill/librdkafka/archive/0.8.5.tar.gz -O librdkafka-0.8.5.tar.gztar zxvf librdkafka-0.8.5.tar.gzcd librdkafka-0.8.5./configuremakemake installcd ..## install rsyslogwget http://www.rsyslog.com/files/download/rsyslog/rsyslog-8.8.0.tar.gz -O rsyslog-8.8.0.tar.gztar zxvf rsyslog-8.8.0.tar.gzexport PKG_CONFIG_PATH=/usr/lib64/pkgconfig:/lib64/pkgconfig/old_executable_path=$(which rsyslogd)executable_dir=$(dirname "$old_executable_path")cd rsyslog-8.8.0./configure --sbindir=$executable_dir --libdir=/usr/lib64 --enable-omkafka --enable-elasticsearchmakemake install## show installation result:new_rsyslog_ver=$(rsyslogd -version |head -n 1 | awk '{print $2}')echo "Old rsyslogd version: "$old_rsyslog_verecho "New rsyslogd version: "$new_rsyslog_verecho "Executable: " $(which rsyslogd)

I have hosted the relevant code on Github:
Https://github.com/garyelephant/rsyslog-scripts

For more information about the omkafka plug-in, see:
Http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html

For more information about the omelasticsearch plug-in, see:
Http://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html

Configuration example:
# /etc/rsyslog.conf# load required module# `imuxsock` provides support for local system logging (e.g. via logger command)module(load="imuxsock") module(load="omkafka")module(load="omelasticsearch")# push to kafkaaction(type="omkafka" topic="your_topic" broker="your_kafka_broker_host_or_ip")# or you can push to elasticsearchaction(type="omelasticsearch" server="your_elasticsearch_host_or_ip" searchIndex="your_elasticsearch_index" searchType="your_elasticsearch_index_type" )

Start rsyslog

rsyslogd -n

Used on another terminalloggerWrite Data to rsyslog

$ logger 'hello world'
References:

Repost this article please indicate the author and the source [Gary's influence] http://garyelephant.me, do not for any commercial purposes!
Author: Gary Gao (garygaowork [at] gmail.com) focuses on the internet, distributed, high-performance, NoSQL

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.