Heka configuration and hekaDistributed backend log Architecture Based on Heka, ElasticSearch, and Kibana
Currently, mainstream backend logs use the standard elk mode (Elasticsearch, Logstash, and Kinaba), which stores, collects, and visualizes logs.
However, our log files are diverse and distributed on different servers and different logs, which will be customized for convenience of further development. As
Recent projects need to use Heka to deal with Redis data, then the problem is, how can it be achieved? The Redis plugin needs to be installed on the Heka.1. Start adding the Heka-redis plugin to the compiler plug-in add-in.cd/usr/local/heka/Vim Cmake/plugin_loader.cmakeAdd_external_plugin (git Https://github.com/victor
time. (The name of this command cannot be named ALL.) the command and parameters are optional. If not specified, an empty target will be created. If WORKING_DIRECTORY is set, this command runs in the specified directory. If it is a relative path, it will be resolved to the build directory corresponding to the current source directory. If COMMENT is set, the value is displayed as information before the command is executed. The DEPENDS parameter can be output of a file and other customized com
Pre-Preparation
Elk Official Website: https://www.elastic.co/, package download and perfect documentation.
Zookeeper Official website: https://zookeeper.apache.org/
Kafka official website: http://kafka.apache.org/documentation.html, package download and perfect documentation.
Flume Official website: https://flume.apache.org/
Heka Official website: https://hekad.readthedocs.io/en/v0.10.0/
The system is a centos6.6,64 bit machine.
Version of the softwa
.
Golang built-in package Encoding/json JSON serialization, regular, reflection, byte-to-string for resource consumption is also relatively high, for the above aspects and the program itself to adjust.
Log storage back-end architectureThe log storage function has Logstash, Heka, Fluentd and other schemes, Logstash based on R Uby, support rich features, but the performance of more complaints; Heka based on
TianTranslated 6 months ago1 Person toptop translation of good Oh!
Despite the collaborative multitasking, your code can still be complex, and it just has the opportunity to maintain a manageable amount of complexity. When control transfer is clear a code reader has at least some visible indications that things may be out of track. There is no clear mark that each new stage is a potential mine: "If this operation is not atomic, what happens at the end?" "Then the space bet
query.
Finally, some of the UI analysis algorithms, using some SQL query statements for a simple, fast query.
Usually from acquisition (logstash/rsyslog/heka/filebeat) to cache-oriented Kafka is a typical wide dependency.
The so-called wide dependency is that each App can be associated with each Broker. At Kafka, each transfer is hashed, and the data is written to each Broker.
A narrow dependency is a process in which each of its FLUENT
development, that is, go allows gogs to be distributed through independent binary, and good support for cross-platform.
From the technical blog finishing, go language-related open source projects:
Tool class:
goclipse: is an Eclipse plugin that can be used to develop the go language
liteide: an IDE designed to develop go
Application class:
Go Read : What can be used to replace Google Reader in the go language
Framework class:
Beego: A powerful web framework
gwk: GWK (Go Web server Kit)
This is a creation in
Article, where the information may have evolved or changed.
Most people who have used the go language recognize it as a very good productivity tool, and some have summed up the following advantages:
Simple deployment
Good concurrency
Good language design
Good execution performance
There are also many successful projects that have been developed using the go language, including: NSQ, Docker, Packer, Skynet, Doozer, Hek
aggregation system
GRAYLOG2-Pluggable log and event Analysis server with alarm options
Heka-Stream processing system, which can be used for log aggregation
Kibana-Visualizing log and timestamp data
Logstash-Tools for managing events and logs
Octopussy-Log management solution (visualization/alarms/reporting)
MonitoringMonitoring software
CACTI-web-based network monitoring and graphics tools
Cabot-monitoring and alerting,
publish-subscribe messaging system; Apache Sqoop: A tool for transferring data between Hadoop and a structured data store; Cloudera morphlines: Help SOLR, HBase, and HDFs complete the ETL framework; Facebook Scribe: Stream log data aggregator; FLUENTD: Tools for capturing events and logs; Google Photon: A distributed computer system that connects multiple streams in real time with high scalability and low latency; Heka: Open source processing s
, Groupcache, Mozilla's Heka, KV and QL lightweight storage systems, Sky performance database.
Outside the open source project, many business projects also use go language development, in the Go user wiki page There are many successful cases, such as:
CloudFlare completely use go to build their distributed DNS service;
There are many systems in the SoundCloud service that are developed using the Go language;
The Ngrok service allows Web de
in Java
Fusion Directory-improved service and corporate directory management based on OPENLDAP
Opendj-opends Branch
OpenDS-Another directory server written in Java
OpenLDAP-Developed by the OpenLDAP project
Log ManagementLog management tools: Collect, Parse, visualize
Elasticsearch-a Lucene-based document store that is used primarily for log indexing, storage, and analysis.
FLUENTD-Log collection and issuance
Flume-Distributed Log collection and aggregation
directory server written in Java
OpenLDAP-Developed by the OpenLDAP project
Log ManagementLog management tools: Collect, Parse, visualize
elasticsearch– a Lucene-based document store, primarily for log indexing, storage, and analysis.
fluentd– Log Collection and issuance
Flume-Distributed Log collection and aggregation system
GRAYLOG2-Pluggable log and event Analysis server with alarm options
Heka-Stream processing s
Objective:Elk is mainly a combination of 3 software, mainly Elasticsearch search engine, Logstash is a log collection log, Kibana real-time analysis to show.[about the Log collection software, such as: Scribe,flume,heka,logstash,chukwa,fluentd, of course rsyslog rsyslog-ng can be collected.About log phone after storage software, such as: Hdfs,cassandra MongoDB, Redis,elasticsearch.About log analysis software such as to use HDFs can write MapReduce ana
Original posted Address:
https://segmentfault.com/a/1190000006880817
A few days ago to view the error log Heka log, found that the fault information too many open files, it is obvious that the number of open file too much.
The first question is how to view the number of files opened by the current process and the maximum number of open files.
Number of files open for current process
Ls/proc/[pid]/fd|wc-l
The maximum number of open files for the curr
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.