This is a creation in Article, where the information may have evolved or changed.
"Editor's note" container technology, as the most remarkable technology of the past two years, has been widely used in various industries, both in the Internet and in traditional industries. As committed to creating a leading financial industry, safe cloud, the introduction of container technology this year, research and development of safe cloud container service platform, bar container technology application to the business, to promote the rapid development of business and technology. The core content of this share is to analyze how to build a safe cloud container platform from user pain point and feature analysis. Divided into 4 parts:
- Container platform Positioning
- Container Platform Design
- Container Platform Architecture
- Container Platform Design Technology
First, positioning
User
First of all, the group's subsidiaries include various categories of the financial industry, each with its own development model, and the need for the underlying computing resources varies.
For the platform, all the requirements are reasonable, so ping an cloud to provide users with different types of computing resources.
Before providing the container, the cloud platform has already stored the cloud host, the cloud storage, now the cloud platform will provide the container service.
Positioning
So we define the container as a virtual-machine container service. Provide a container service that complies with the peace infrastructure and user self-service.
Computing Service "Three Musketeers", meet different kinds of computing needs: Physical machine + virtual machine + container
As a lightweight and convenient method of calculation, container enlistment needs to develop its advantages.
Second, the design of the platform
Platform Construction
1. For multi-tenancy
"Internal public cloud", need to meet the needs of multi-tenant security isolation, have sufficient permissions to control
Supports multiple computing methods for interoperability
2. Integrated User Experience
- Meet the characteristics of cloud services to streamline the delivery process
- Take into account user habits and existing processes
- Take advantage of containers for micro-innovation
Here are my products:
Platform Design-conceptual model
- Tenant: A tenant concept that corresponds to a cloud portal can be understood as a department or a product development group.
- Environment: That is, the carrier of the container, the environment contains several hosts, the environment of the host jointly bear the operation of the container.
- Host: Can only belong to a specified environment and cannot be deployed across environments.
- Application: Similar to the system concept, the stack concept of the corresponding container
- Orchestration: Docker Compose
- Service: Docker Service
- Container: Docker Container
- Image: Docker image
Platform Design-platform features
1. Container Environment
Tenants have a single set of container services in one data center
Users can create multiple sets of container environments based on their requirements and add their own compute nodes to the environment to enable elastic expansion of the underlying resources.
The environment is completely isolated from resources, services, etc., and is only associated with the management state.
2. Application
Supports Docker Compose V1 orchestration creation.
Provides common orchestration scenarios, one-click Deployment applications, Application and orchestration transfer, and personal orchestration to the App Store.
3. Service
Supports configuration such as service connections, port mappings, environment variables, directory mounts, and more.
Detailed Service Management: Service information, service events, service configuration.
Support shell, file upload, image production and other extended functions.
4. Image Management
Mirror Mall: A variety of official images are available for all tenants.
Public warehouses: Each tenant has its own public warehouse, and users in the tenant share the intra-tenant image.
Personal Image: Supports submitting containers as mirrors, downloading official and public images.
Supports multi-zone distributed image management.
These are the four functions we provide for the container field.
III. Container Deployment Architecture
Let's look at our current platform system architecture as a whole:
Each of these areas is the company's network architecture for the financial industry.
Specific considerations are as follows:
First, the portal is deployed in a cloud zone to ensure that all tenants can use the container service through the portal.
also, our mirrored mall is also deployed in the cloud.
Deploy one or more sets of container scheduling systems in each data center for different regions and user needs.
As in the development area, we provide a set of orchestration systems where all tenants use a set of container services.
In a production environment, we create a set of orchestration systems for each tenant so that the tenants are completely isolated from each other.
In addition, the container platform will interface with the monitoring platform and Configuration Management Center.
Each of these areas is the network architecture that Ping An cloud is setting up for the financial industry.
Describes the deployment components, and then details the capabilities of each component and the technologies that are used.
Cloud Precinct (external)
CaaS Portal is our core and user interface for direct contact. The specific introduction is the function of the previous chapter of the interface.
CaaS Portal Backend Database We use the MySQL cluster, using the Galera cluster clustering technology.
Pub Hub is our mirror Mall for safe users, because of the network and specifications, we have to set up a mirror hub. Download the most important image of the Docker hub to the mall.
Public service Area (cloud Platform resource Area, internal)
Docker server: We encapsulate a set of Java-implemented Docker APIs based on the remote Docker API and deploy it as a Docker server Agent.
Used to execute Docker commands, such as submitting a mirror to a container, to obtain the mirrored data.
Orchestration: We use the open source container orchestration platform rancher to manage the container by invoking the API.
Registry Mirror: To not allow all Docker hosts under a tenant VPC to access the pub Hub, it also reduces the transmission and security between networks.
We built a set of registry in each data center to communicate with pub hub. When the user needs to create the container.
In addition, we configure the tool to use the ansible, which mainly executes the corresponding command on the Docker host. For example, the VM becomes Docker host and is added to the environment.
We wrap the ansible in a restful form.
Iv. Container Platform Technology
The overall architecture is shared, followed by a description of the technologies involved in the container service and our solutions.
The technologies involved in the container area include:
- Container Network
- Container storage
- Container Log
- Container monitoring
- Image Management
The entire container technology is also about these 5 chunks:
Container Network
First introduce the container network history, I will forget the container development into 3 stages:
First Stage:
Container's oldest network model, available in 4 ways. Bridge bridging mode, which uses the host IP and maps the container port to the port of the host. The disadvantage is that a large port usage can cause conflicts.
Host Mode:With the host IP, the disadvantage is that a container uses a port and no other container can use the port anymore.
Container Way:The network is shared with other containers on the host.
None Way:Do not set up the network, need to manually configure,
Phase II:
In order to solve the above problems, there are a lot of programs, more famous in the following three kinds, the three are always to solve the cross-host communication, the largest extent to compensate for the network model of the container itself shortcomings.
Technically, these container technologies can be categorized into two technical genres: tunnel mode and routing mode.
Ipsec,fannel belongs to tunnel mode, while Calico belongs to route mode.
Phase III:
Later, Docker and Google specified standards for their container network models.
The standard that Docker introduces is the company, which can manage the network model directly through the Docker command.
After introducing the network knowledge, we look at CaaS select the network.
external:Container Bridge with Port mapping
internal:Implementation of IPSec tunneling between containers (directly using the Orchestration Framework rancher)
If a container needs to be serviced externally, the service can be exposed by port mapping and by connecting to the VM in which it resides.
If the container does not need to provide services to the outside, only need to provide services within the same application, then use the IPSec approach, so as to avoid wasting a multi-port.
A private forget between the container and the container, only the container and the container can be accessed. Each container has a private network address: 10.42. Network segment.
Below I am a Tomcat container we created through the container platform, and I map the container to the host so that it can be accessed from outside.
Among them, the external port occupancy problem, we unified distribution and recycling.
You can see the container and the host IP239, as well as the container IP10.42.243.155. Here is the integrated use of the ports silver and IPSec.
Container storage
Goal: Extremely fast creation speed, minimal storage resource consumption, and ease of container migration
Technology: tiering and write-time replication cow
Implementation: Aufs,device MAPPER,ZFS,OVERLAYFS
Our container service is currently running on CentOS and no other operating system has been used. So we chose Devicemapper as the container storage driver.
Below is the maturity of all the storage drivers provided by Docker. It can be seen that the Aufs and Devicemapper are now more mature.
Currently only these two are the official production-ready of the proposal. Our container platform selects the Devicemapper.
Docker storage System selection development compared to the earlier mature devicemapper, on the one hand because the technology has been incorporated into the LINXU core, stable and mature, on the other hand we choose the VM is Ceotos, it uses the default is Devicemapper.
The Devicemapper layer uses the cloud disk directly as the pool, using LVM management.
In addition to container and mirrored storage, another piece of storage is overlooked: Application data storage, including configuration files, our scenario is: using the volume interface or direct volume mapping
This resolves the container to migrate automatically.
Log
- Container Service Platform log: Local + Cloud Platform Elk Log Service
- Container self-running log: On-premises Cloud disk + Cloud Platform Elk Log Service
- In-container application (Business Party) logs: Business self-planning, catalog mounts already available
Container monitoring
This monitoring, we rely on the original monitoring of the cloud platform, additional to the container-specific monitoring.
1, host monitoring
2. Container monitoring
Self-developed scripts provide performance monitoring of the container itself (cpu/mem/network/storage), monitoring platform timing acquisition, and being able to view it on the portal.
3, the specific middleware monitoring
Provides performance monitoring of common middleware (weblogic/tomcat/nginx, etc.), scripts for middleware mirroring, middleware monitoring programs integrated into Docker images, and container start-up to instantly escalate performance data to the monitoring platform without any external intervention. This monitoring can be viewed in the monitoring platform.
The monitoring platform is developed by another team in the department. Currently using Zabbix/open-falcon.
Image Management
For image management, we evolve from cross-region distribution, single-node, and two-node.
1, image management using distribution, front-end settings lvs+keepalived for load balancing and high availability, mirrored storage using DNAs.
2, Docker server is our own development of service components, each region (such as Shenzhen, Shanghai) to deploy a set of docking registry in the region. On the one hand, listen to the registry event of the current zone and initiate the synchronization operation. On the other hand: Executes the Dockcer command, which manages all the machines running the container in the zone.
This feature has just recently been completed. There are only three ways to manage distributed image management across regions:
- Proactively perform synchronization, such as calling the Docker remote API to perform Docker Push/docker pull.
- Event monitoring, using Registry's notification monitoring.
- The underlying storage is synchronized.
We combine three solutions, combining the 1th and the 2nd in the practical way.
Q&a
Q: Can I communicate between container clusters across regions?
A: Currently, cross-region machines cannot communicate. Machines across regions are completely isolated from tenants. If the tenant needs to communicate, open the wall for the specific machine.
Q: What questions do you encounter with Devicemapper?
A: The most commonly encountered is device busy. Causes the container to not be deleted. Human intervention is required at this time.
Q: If the image management is only based on distribution, how is permissions and multi-tenancy management implemented?
A: Permissions into the platform management, by our own implementation. Mirrors between tenants are not shareable. Shareable and private within the tenant.
The mirrors between the tenants must be published, and the platform auditors can be shared between each region and the individual tenants.
Q: What language is monitoring developed, is it self-developed or open-source?
A: monitoring, I have done some monitoring data collection for the mirror and container, and then provided to the monitoring platform. The monitoring platform is developed by a different group of departments. The current development is based on Zabbix.
Q: Excuse me: 1. Dockerhub only one instance? How to ensure high availability? 2. What is the synchronization between Docker servers? Docker event?
A: All areas, including pub and Mirror, are Keepalive+lvs built. A dual registry+ shared NAS is currently used. And these two registry VMs are located across AZ. That is, a registry outage does not affect another offer at all. While the NAS is a dual-write Nas, our side of the storage group provides the DNAs service. Backup policies can also be defined according to their own.
q:pub Hub How to get the Docker hub image, is it automatic maintenance now?
A: No, from the Docker hub into the company's official image warehouse, are manual maintenance synchronization. We need to do special processing for some mirrors.
Q:IPSEC performance legend is not very good, you have encountered the problem of the network? What about the Rancher IPSec network performance? Does the container involve elastic scaling? Rancher for network communication between containers, why are there no similar components selected?
A: These 3 questions are answered in a unified answer. IPSec is currently used. Since the number of containers currently docked to each rancher is not particularly large, network problems have not been generated. In addition, the external network to go is the cloud host network.
There are other components that have been considered, due to the previous start-up phase, all directly taken rancher provided. Recently, rancher has provided support for the MLM programme. The corresponding components are currently being studied.
Q: One of the solutions I can think of is the user system of the tenant and cloud platform of the cloud platform, and the mapping of many to 1, when the audit pass, use different cloud platform users, so the permissions are not the same. Is that right?
A: The current container service is only open to the internal company, directly compatible with cloud platform tenants.
Q: What is the kernel version? With a 4.9 plan? Is Ping an OpenStack? With Magnum?
A: Using CentOS 7.2, kernel version is 3.10.xxx, ping An cloud uses cloud Foundry, Ping An cloud uses a variety of virtual machines, VMware and KVM are the main.
Q: Does the container run on a physical or virtual machine, and is the container's operating system owned by the tenant or operator? Who decides the allocation rules for containers? Who decides how many containers a host runs?
A: The current resource constraints or to rancher to manage, our rancher planning is: let rancher management resources, allocation of containers. The page provides the resource parameters that Docker currently supports.
The operating system where the container resides is owned by the tenant. Tenants can add their own cloud host to the container environment.
Q: Is the system currently used internally? Has security been considered?
A: Security: Currently for the internal company, using the company AD Authentication login. The compute resource limits are tenant-isolated. Mirroring this piece of security using host trust.
Q: Many business logs are not directly exported to stdout, how are these business logs collected? What is the method for collecting container logs? How big is the container size? Will there be a warning when the acquisition is interrupted? What are some of the more effective reporting data when you enter ES?
A: The log needs to be divided into three types:
Container Service Platform log: Local + Cloud Platform Elk Log Service
Container self-running log: On-premises Cloud disk + Cloud Platform Elk Log Service
In-container application logs: Business self-planning, directory mount already available
This side of the platform currently focuses on the top two logs.
Log this block will also be based on the importance of the choice to report police.
Business log: We offer solutions.
Log management for business parties: one of the important aspects of the need to help the business parties to containerized the system is log management.
Current adoption Scenario: Let the business parties to the log container, so that the log does not hit the container itself on the machine.
The above content is organized according to the December 27, 2016 night group sharing content. Share people
Zhu Yuhua, head of container service platform of Ping An technology cloud platform division. In 14, he worked as a soft-to-application architect for Ping An technology architect, a cloud platform development engineer, and is currently responsible for the architecture design of container service platform.。 Dockone Weekly will organize the technology to share, welcome interested students add: Liyingjiesz, into group participation, you want to listen to the topic or want to share the topic can give us a message.
