Kubernetes improvement and optimization to support large-scale public cloud (1)

Kubernetes is designed to implement a private container cloud, but the container as a public cloud, also requires a management platform, in Swarm,mesos,kubernetes, based on Kubernetes has gradually become the container orchestration of the hottest and most mainstream platform, NetEase realizes its own container public cloud based on Kubernetes, in this process, needs to make certain improvement and optimization to kubernetes.

The schema is as follows:

NetEase developed its own container service platform, combining OpenStack's IaaS layer and kubernetes container layer depth to achieve a complete public cloud system. As you can see, the container service platform dispatches the OpenStack compute service Nova to create the KVM virtual machine, then calls Cinder to create the cloud disk on Mount, call Neturon to create and connect the network, and then call Kubernetes to create the container. You can also call NLB to mount the load balancer.

First, the OpenStack architecture is complex

Before the container platform, NetEase's IaaS layer was based on the OpenStack architecture. It is said that OpenStack is too complex to be an architecture diagram of OpenStack.

OpenStack mainly includes the following modules:

• Safety Authentication Module Keystone

• Compute Virtualization Module Nova

• Storage Virtualization Module Cinder

• Network Virtualization Module Neutron

• Image Management Module Glance

• Object Storage Module Swift

Each of these modules contains many sub-modules, most of which include API modules, dispatch modules, and specific working modules.

Second, OpenStack the process of creating virtual machines is complex

The flow of OpenStack to create a virtual machine is complex, and here's a brief summary of the key points.

The first: AAA, that is, we often say authentication,authorization,account.

The so-called authentication certification, is to verify that I am not me, authorization authentication is audit, although I am me, but I do not have the right to do this thing.

Authentication generally there are two ways, one is the symmetric encryption method, that is, with a token, the client and the server with this token for encryption and decryption, a non-symmetric encryption method, that is, the use of PKI, the use of certificate way.

AWS is also available in either of these ways.

In addition authorization, the role based access control is commonly used.

There are users, roles, and tenants of the concept.

For example, AWS has

Second: Nova-api Accept the request

Here are two things you can do, rate limit, call I can't be too frequent, quota, control how many resources each tenant can create.

Third: Nova-scheduler to dispatch

Scheduling is divided into two processes, one is filtering, first will not meet the requirements of the host filter out, one is weighting, the remaining based on the use of the host to score rankings, select a machine.

IV: Nova-compute really working people receive a request, call Libvirt to create a virtual machine

V: Libvirt is the real tool to create virtual machines, first download the virtual machine image

Sixth: Libvirt start to define KVM startup parameters

Seventh: Libvirt start to create network devices for KVM

Eighth: Libvirt start KVM, where cgroup is typically used to control the use of KVM resources

Nineth: Call cinder to create the storage for the virtual machine, and the backend typically uses ceph

Want to know kubernetes people are not see here already bored, not speak kubernetes? How do you tell so much about OpenStack?

Let's take another look at the graph, this is the Knowledge Atlas of AWS creating virtual machines, isn't it a lot of similar concepts?

A lot of technology discovery technology development is too fast, from virtualization, to OpenStack, to Docker, to kubernetes and so on, how to learn, in fact, in-depth understanding will find that the basic technology is very similar, including the next analysis of the kubernetes.

Third, the structure of kubernetes is relatively simple

A lot of people like Docker, and Docker platform, is very simple docker, without openstack so complex concept, it is easy to start an nginx demo.

As a container management platform, the architecture of Kubernetes is also relatively simple.

Customer requests come in, first into the API layer, equivalent to NOVA-API, first to authenticate and authentication (authentication and authorization), the equivalent of Keystone do things.

The created object is then stored in the ETCD, and if OpenStack is inside the database.

Then the scheduler, the object is dispatched to a machine, the equivalent of Nova-scheduler to do things.

Then the Kubelet on each machine was really working and found himself being dispatched to create a container on its own machine, equivalent to Nova-compute.

Kubelet you create a container, you first download the container image, and nova-compute also download the mirror of the virtual machine.

Nova-compute to invoke the Docker interface to create the container, equivalent to the Libvirt created by the Nova-compute call Kvm,docker true isolation using CGROUP,KVM also use Cgroup, Docker also uses NAMESPACE,KVM's network configuration to use namespace.

Docker was created, needed to configure the network for Docker, configure storage, and Libvirt did it.

Iv. Kubernetes The process of creating pods and service

• The client invokes the API interface to create the pod.

• Api-server creates an object in the pod, which is stored in the ETCD.

• Scheduler constantly through the api-server to see which pods need to be dispatched, then dispatched to return the dispatch results to Api-server

• Api-server writes the Scheduler schedule results to ETCD.

• Kubelet constantly check to see if there are pods that can dispatch to their machines, and some call Docker's interface to create a container.

• The client invokes the API interface to create the service.

• Api-server creates a service object to write to Etcd.

• The controller constantly scans the pod for the service.

• The controller calls Api-server to create the corresponding access endpoint endpoint.

• Api-server writes the endpoint object to Etcd.

• Proxy is constantly discovering that there are no forwarding rules that can be placed on top of itself, and if so, create a socket listener port and create a corresponding iptables rule.

Five, kubernetes nothing?

Kubernetes looks a lot simpler than OpenStack, but it lacks a lot of functionality.

• There is no perfect tenant management module, tenant isolation is not good, do you need a service like Keystone?

• Is it necessary to manage the image, do not need a service like glance?

• Where is the image stored, does it require a service for object storage, similar to Swift?

• Kubernetes itself no matter the network, need to through the plug-in, Network and SDN who to manage?

• The kubernetes itself, regardless of the storage, needs to be done via plug-ins, and most of the storage options are done via Ceph.

However, if you want to make a public cloud, at least take care of the above section, if you add these to the equivalent of a kubernetes based on the re-creation of an OpenStack, why do you want to repeat the wheel? So we chose OpenStack and kubernetes for a deep fusion solution.

Today, the plane was late, originally a day, should have written yesterday had to finish the morning.

Next, the solution for OpenStack and kubernetes Fusion is resolved.

In fact, there are more questions as public clouds:

• Network two-time virtualization issues

• Issues with the public and floating IP

• A kubernetes cluster or multiple kubernetes clusters?

• Kubernetes Cluster If you do a large scale?

• Wait, wait.

will also be elaborated in the next series of articles

Kubernetes improvement and optimization to support large-scale public cloud (1)