Use Netsil to monitor microservices on Kubernetes

Ubernetes is the King of container orchestration and scheduling, it beats the competitors Docker swarm and Apache Mesos, opens the bright future, microservices can be self-repairing, can be automatically expanded, can cross zone, Region even across cloud providers for federate. In this new era of cloud-native applications, it is becoming increasingly important to have a simple insight into how services interact with each other, which can be different from a haystack-wide search for a specific cause of performance problems.

We spent some time studying Netsil and packaging its solutions into native kubernetes Deployment. The Netsil application, Application Operations Center (AOC, Application Operations Centre), helps users observe and collect profiling data for microservices applications running across kubernetes clusters. The service itself is unknowable, because it is on the network to determine how it actually works. Over time, and in real time, it learns and discovers the user's environment, helping users build up SLA indicators, alarms, and so on.

Let's go

First you need a kubernetes cluster. I use Stackpoint.io to quickly create a cluster. Create a cluster on any major vendor, such as AWS,GCE or Azure. You need to make sure that you choose a large enough configuration for your primary node-this is where all collectors will send the data, which can be very expensive on the network, processor, and memory. Worker nodes can be any configuration, as long as they meet the needs of microservices applications. In my example, I used a large instance configuration because I would push a variety of services into this environment.

In our example, a cluster was built using 3 instances of N1 standard 4, which were exposed through the Haproxy ingress controller, were self-discovered, and were registered with the AOC service when they were deployed. We are able to access the AOC dashboard using the public VIP of the cluster.

Before you begin

Install some services on an empty cluster that only runs the Kubernetes service, using sock shop, a microservices reference program developed by Weaveworks. This helps to simulate a real environment. Sock Shop uses 14 different services, which are the complexities that many enterprise applications can achieve. AOC is now added to our environment.

Detailed information about the sock shop is available here. Pushing it to the environment is simple, just need to run the following command after cloning the repo:

Kubectl Apply -F Deploy/kubernetes/manifests     

Then check to see if the pod is online:

$ kubectlGetPods--Namespace=DefaultNAME Ready STATUS Restarts Agecart-3694116665-Eccpp1/1Running055mCart-Db-2305146297-U30g81/1Running055mCatalogue-11453786-Lkslj1/1Running055mCatalogue-Db-393939662-Bn7uc1/1Running055mFront-End-3820830240–01e6t1/1Running055mOrders-3498886496-Z8jun1/1Running055mOrders-Db-1775353731-U7dmf1/1Running055mPayment-3012088042-Vbfhw1/1Running 055mQueue-Master-936560853-Ocmxi1/1Running055mRabbitmq-1897447621–2ij041/1Running055mShipping-1232389217-b278a1/1Running055mSpc-Balancer-Biilo1/1Running 0 1huser-3090014237–196pv 1/1 Running 0  55mUser-db-1338754314-exyou 1/1 Running  0 55m                

Let's start watching.

We already have a running Kubernetes 1.4 cluster, and have installed the sock shop app, so what is the starting learning environment? Can we know when a stock buyer is having problems?

You need to run the following commands on all hosts before you deploy AOC. This command helps to avoid a known flannel and kube-proxy competition problem.

Iptables -t Nat -I postrouting -o flannel.  1 -s host-private-IP -J Masquerade      

Replace with a private IP for each host host-private-ip . After completion, clone AOC Kubernetes repo from GitHub:

git clone https://github.com/netsil/netsil-kube.git 

and push it to kubernetes using the following single command:

Kubectl Apply -f netsil.  Yml   

Make sure the pod and service are online. The AOC container may take some time, but the collector will be started and the data in the queue will be pushed in, as they are already discovering your environment.

$ kubectlGetPo,Svc—Namespace=Netsilname Ready STATUS Restarts Agecollector-7wpaa1/1Running01hCollector-9o6k41/1Running01hCollector-Rzekv1/1Running04mNetsil-vjf5f1/1Running0 1hname cluster- ip external-ip port ( s)  agenetsil 10.200 126.143 <nodes> 443/tcp,2001 /tcp,2003/ tcp,2003/udp < Span class= "lit" >1h               

The AOC topology has two main components. The first is a pod that runs as part of a replication controller with a single copy. It runs the AOC dashboard and data collection platform. The second component is the daemonset of the AOC collector. It tells Kubernetes to run a pod with a collector container on all nodes of the environment. These collectors are configured to send information to the AOC pod.

Generate traffic

We will use more tools from the sock shop to simulate the shopping behavior on our website. This allows us to see how AOC learns traffic patterns and our common topologies.

You need to know the front-end IP addresses and ports that sock shop listens to and runs:

Docker run Weaveworksdemos/load-test -H $frontend-IP[:$port]  -R  -C 2             

As the load-test runs, you can start to see that AOC is lit up as the data gets:

Since AOC is deployed as a daemonset, if any pod is destroyed and re-dispatched elsewhere, AOC can continue to observe the topology and change as the kubernetes changes.

One of the reasons I like AOC is that the deployment is organized through services, and I am able to observe the environment in real time, and begin to drill down into different metrics to build service level alerts for things that might affect the customer. So, when the environment turns red, I get alerts to know that a service is in a state of emergency, such as the credit card and address endpoint in the sock shop.

I can even drill into the dashboard to know what pods and containers are under the greatest pressure. In this example, the largest network pressure container is the flannel Pod. This allows us to see which of the busiest services are, and to help us rethink the way we deploy or distribute them in kubernetes.

Summarize

Netsil's AOC is a great tool to help users observe the environment in real time and update as usage patterns change. Users can tap historical data and add alerts. The application will automatically expand as more nodes are added, and the new node will launch a collector on top of the line, so that the user can get all the data from the node to the destruction.

If you want to use Application Operations Center in your kubernetes environment, just download the manifests here. You can learn Netsil and application Operations Center at http://netsil.com.

Use Netsil to monitor microservices on Kubernetes