Pits Guide to Kubernetes fluentd+elasticsearch+kibana log setup

Source: Internet
Author: User
Tags kibana fluentd

After the initial completion of the Kubernetes cluster architecture, by building some monitoring components, we have been able to achieve

    • Graphical monitoring of status information and resource conditions for each node,pod
    • Scaling and scaling of replicateset through scale
    • View the run log for each pod by kubectl logs or dashboard

However, the scale of the nodes in the distributed architecture is often very large, a typical production environment may have dozens of hundred minion nodes, in this case it is necessary to establish a set of centralized log monitoring and management system, in my early thinking, I also want to implement WebLogic log output to shared storage by Volumn plug-in to storage, but the problem with this approach is:

    • We extend the Docker service by WebLogic Single-domain mode, which means that all log paths and names are consistent, i.e. they are stored in a uniform path to the container node of the pod (/u01/oracle/user_projects/ Domains/base_domain/servers/adminserver/logs/adminserver.log), there is a file conflict problem if you map to storage through Volumnmount.
    • Unable to get information about pods and container
    • Unable to get run information for other nodes in the cluster

It is therefore still necessary to look for platform-level architecture scenarios. In Kubernetes's official documentation, https://kubernetes.io/docs/concepts/cluster-administration/logging/

Kubernetes gives several log schemes, and gives the reference architecture of Cluster-level logging:

Kubernetes recommends this node-level logging-agent and provides two of them, one for Google Cloud platform Stackdriver Logging, and the other for Elasticsearch, Both are using FLUENTD as agents that run on nodes (log agents)

Using A node-level logging agent is the most common and encouraged approach for a Kubernetes cluster, because it creates O Nly one agent per node, and it doesn ' t require any changes to the applications running on the node. However, Node-level logging only works for Applications ' standard outputand standard error.

Kubernetes doesn ' t specify a logging agent, but the optional logging agents is packaged with the Kubernetes Release:stac Kdriver Logging for use with Google Cloud Platform, and Elasticsearch. You can find more information and instructions in the dedicated documents. Both use FLUENTD with custom configuration as a agent on the node.

Okay, here's our pits guide.

1. Preparatory work

    • The Kubernetes code in GitHub is planted down to master local.
git clone https://github.com/kubernetes/kubernetes
    • Configure ServiceAccount, this is because after the download of FLUENTD images need to use SSL to connect to the API Server, if not ready to modify and generate a new images, or need to be configured, configuration Guide reference
http://www.cnblogs.com/ericnie/p/6894688.html
    • Configuring the Dns,kibana component requires DNS to find the Elasticsearch-logging service, and if you do not configure DNS, you need to modify the address in the Kibana-controller.yaml to a fixed service IP. Configuration Guide Reference
http://www.cnblogs.com/ericnie/p/6897142.html

Pits Guide to Kubernetes fluentd+elasticsearch+kibana log setup

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.