This is a creation in Article, where the information may have evolved or changed.
"Editor's words" This share is mainly zookeeper, Redis, Kafka, MongoDB and other applications containerized in the Kubernetes platform above practice. The problems of application in integration are analyzed from the aspects of computing, network and storage, and the application of some traditional applications in the process of container design is two times. This paper introduces the requirements and limitations of application Docker, then introduces the basic platform, the overall environment includes kubernetes and ECP, then introduces the practice of specific applications such as zookeeper in integration, and finally introduces the two development that some open source applications have designed in the containerized process.
Container is a lightweight virtualization technology with continuous integration, versioning, portability, isolation and security, and more and more applications are running in containers. But there are flaws, not all scenarios are suitable for such as high-performance computing, already full-load operation of the application is not necessary for virtualization, some of the requirements of the operating environment such as IO is not suitable for containerized such as Oracle database.
Containers provide an independent environment for applications, not a complete set of operating systems like virtual machines, which is the biggest difference between containers and VMS, so when users are using virtualization such as KVM and VMware, there is little concern that applications can run into virtual machines. Typically only performance tuning is done for virtualized environments, such as KVM CPU bindings, memory balloons, and giant pages. But the application of Docker is more complex, in addition to solve the virtual machine exists in the computation, storage, network problems, but also need to solve the problem of Docker itself, such as: Docker relative to the virtual machine fragility, application-dependent system calls in Docker do not limit, There are also issues such as Docker image authoring and post-Docker logging and performance acquisition.
Docker restricts CPU and memory usage by Cgroup, CPU qualification is usually set to weight, does not allow a container to get a 1Ghz CPU, of course, you can also use Cpuset to bind to the specified physical core, memory usage not only to set the memory size but also to limit the size of the swap partition , which is important, such as MongoDB's large use of swap partitions when there is insufficient storage.
The relatively short life cycle of Docker determines that no persistent storage is used, the container is hung, and the stored data is lost accordingly. If the data needs to be persisted such as MySQL's dbdata data, you need to export the data to an external storage device, which can be supported by mounting NFS or GLUSTERFS or AWS storage, and the back-end storage of the Kubernetes PV can already support the above file system. Users can also implement Docker volume plugin based on different storage, which is popular with Rancher/convoy and Flocker, which supports data migration, which lays the data foundation for container migration.
Container applications rely on each other: Web applications Access databases, API interface calls, and so on, so the network must be interoperable and multi-tenant isolated, with the current Docker network having two major camps, CNM, and MLM. Which Calico can be considered both camps, it can be integrated with the kubernetes, compared to the overlay network in the network transmission process still has a clear advantage, as shown, reduced packet encapsulation improves network transmission and packet processing efficiency.
Docker OVS SDN is also a great design idea for networking between containers and containers, as well as containers and hosts, by enabling Docker's network drivers interface to hook up containers to OVS through Ryu, Floodlight or Opendaylight and other SDN controllers dynamically release the flow meter to achieve network interoperability, and then combine iptables to achieve network isolation. If you want to achieve a sophomore layer across the computer network interoperability, you can assemble Vxlan and other overlay technology.
Sky Cloud Skyform ECP platform is a container management platform based on Kubernetes Calico, which has a complete container management platform including log collection analysis, performance monitoring and image management. Some of the relevant knowledge related to Docker is described above, and the integration of specific applications on the Kubernetes platform is described below. I put the application integration difficulty into three levels: the most easy is the application itself has service discovery mechanism such as Dubbo, natural support auto-scaling capacity, then the Kubernetes integration will be very simple; most applications need to modify the configuration to be able to run inside the container, The most difficult is the need to two development of the application or two development of the platform application.
ZooKeeper
Zookeeper is a high-performance, highly available distributed coordination service. The entire zookeeper cluster provides 2181 service ports, data synchronization between nodes, and 2888 and 3888 ports used respectively by the main selection. Each zookeeper node requires a different node communication, so a fixed IP address is required, and the virtual IP provided by the Kubernetes service (if DNS is configured using the cluster) communicates with each other, as shown in the following structure:
Yaml files are as follows
Storage uses NFS shared storage so that the zookeeper container can recover data after it is rebuilt.
Kafka
Kafka, a distributed messaging system developed by LinkedIn, is written in Scala and is widely used for horizontal scaling and high throughput rates. At present, more and more open-source distributed processing systems such as Cloudera, Apache Storm, and spark support integration with Kafka. The entire structure diagram is as follows:
The Yaml file is as follows, there are three places to be aware of:
- Each Kafka broker has a unique broker_id within the cluster, so the entire Kafka cluster cannot be placed into an RC, and each broker must be defined by a different RC;
- Kafka relies on zookeeper, where we pass the domain name to the container in the way of environment variables;
- There is no persistent storage configured for Kafka because each topic contains one or more partition, and it is very fast if a single broker outage does not affect the cluster and automatically synchronizes the data when the broker is rebuilt.
Redis
Redis is an open-source (BSD-licensed), in-memory data structure storage system that can be used as a database, cache, and message middleware. In the Redis container testing process, there are several performance issues to note, the first is the maximum redis memory, if not set may appear to run out of memory phenomenon, the second is the data TTL time setting and cleanup keys policy, these are to avoid low memory when using swap partition caused performance degradation, The third is that data persistence uses RBD, and RBD is more suitable for snapshot recovery than for large file aof. If you have a small business scenario, you can use the Redis master-slave structure, as shown in:
Resis Master Yaml files are as follows, where for data migration recovery and reconstruction using Glusterfs storage, of course you can also switch to other storage, a single master-slave cluster may not be able to host too many requests, can be through the current more mature a scheme twemproxy, Data fragmentation via proxies can also achieve a good result:
Currently we are testing the Redis 3.x cluster feature, each redis is a service (slave default does not accept read and write, need to perform readonly to read), eliminating proxy agent to improve efficiency. This cluster scheme is built with the CREATE subcommand of REDIS-TRIB.RB, which can operate the data in the cluster by configuring all service addresses on the client, but the entire operation process has two parts that must be completed by hand, yet to be perfected:
- The entire cluster creation redis-trib.rb is accomplished by hand, without service discovery and automation;
- Add dot you must manually perform the REDIS-TRIB.RB reshard allocation slot.
Mongodb
MongoDB is written in C language, is a distributed file storage based open source database system. MongoDB has deprecated the use of master and slave, and clusters can select replica Sets or shards. Each MongoDB shown is a service that makes up a replica set of clusters, similar to MongoDB, which requires higher disk IO to use SSD acceleration, which can be created with the label selection to improve performance on SSD machines.
In the case of a MongoDB shard cluster, the structure is as follows:
Each of the CONFIGSVR configuration services and the MONGOs routing service provides a fixed provider through the service. MONGOs through CONFIGSVR domain name configuration. The feature test for this shard has passed, but the performance test is in progress, with the script dynamic configuration of the service discovery to turn the replica set into shards.
Some of the above applications are passed through the container outside the configuration parameters or startup scripts to modify the configuration of the application, but some applications can not be completed in this way, it must be done by the application or the transformation of the platform, such as the IBM task scheduling platform Openlava, which through the/proc/meminfo and /proc/cpuinfo gets the memory and Cpu,docker this part is mounted by the host proc file system, causing the application to read in the container is the host's CPU and memory, at this time by modifying the Openlava code to make it from/sys/fs/cgroup/cpu/ Cpu.shares gets the number of CPU cores from/sys/fs/cgroup/memory/memory.limit_in_byte to get the memory value, some applications (multiple pods) need to share IPC, this part of the function kubernetes is not, The need for platform transformation to meet the application needs, as well as application needs to be based on Io to dispatch, these are our ECP platform based on Kubernetes added features.
Q&a
Q: How to build a distributed service framework under the Kubernetes architecture, such as Dubbo need to register host address and exposed port to zookeeper, this host address and exposed port how do you deal with, that is, the application in the container how to get the Docker host address?
A:dubbo service does not need to expose the host's IP and address, just need to know the container address. These services are registered to zookeeper above, through the zookeeper to complete the service discovery, Kubernetes can be based on the exposed port scheduling, of course, some applications require the container to obtain the host's IP address, this block we made some changes to the kubernetes, You can dynamically inject the host name information such as the host IP into the container's environment variables.
Q:ECP The positioning and resolution of the target compared with the current people in the traditional cloud platform solution in terms of?
A:ECP product positioning is a complete container solution, from container lifecycle management, resource monitoring to log analysis processing, compared with the cloud Platform solution management object is no longer a virtual machine, but a container, object oriented is the service.
Q: The resource performance monitoring of the container itself, is the use of cadvisor+heapster, how to maintain the pod restart (after restarting the pod name changes) after the data continuity, thank you.
A: Resource monitoring is not HEAPSTER,ECP resource monitoring is the use of cadvisor+ our own research and development of the acquisition of Agent+ceilometer+mongodb+hbase and other technologies. Reuse of our technology in the production of CMP products, RC pod after the reconstruction will be renamed, this block for the continuity of single pod data is not considered, we consider the RC as a whole.
Q: How does the load balancing of your external services work? is the service directly with Kubernetes?
A: External load balancing at this stage of the Kubernetes service, taking into account the performance loss caused by iptables, we will consider the use of other programs, directly within the cluster to forward traffic to the pod.
Q: How is the Kubernetes container and the application monitoring running in it done? Can you introduce me? Thank you!
A:ECP's resource monitoring is a collection of agent+ceilometer+mongodb+hbase and other technologies developed by cadvisor+. Reuse of our technology in the production of CMP products. It is simple to use cadvisor to capture raw data, and then through Ceilometer persistence, to provide real-time data query, alarm and other functions. Data is periodically dumped into hbase for historical data analysis.
Q: Is there an open source implementation for multi-tenant and user quotas based on kubernetes?
A: There are fewer open source solutions, and our ECP products are multi-tenant management combined with Keystone and tenant quota management.
The above content is organized according to the August 9, 2016 night group sharing content. Share people
Chen Xiaoyu, Beijing Tian Yun Rong Chong Software Technology Co., Ltd., Research and Development engineers, many years of cloud computing and Distributed system development experience, Cloudstack, OpenStack and other cloud management systems have in-depth understanding. Long-term commitment to the application of Docker, Docker clustering solution design and development。 Dockone Weekly will organize the technology to share, welcome interested students add: Liyingjiesz, into group participation, you want to listen to the topic or want to share the topic can give us a message.
