This is a creation in Article, where the information may have evolved or changed.
"Editor's word"As a "spoiler", containerized technology is bound to face the challenge of adapting to the existing architecture of the company, this article will introduce you to 360 how to let Docker landed. Mainly includes three aspects: first, combining the company's business characteristics, how to make Docker adapt to the existing technology structure, complete the online environment rapid deployment capacity expansion; second, "make the product less expensive", use Docker to build PAAs environment to accelerate small and medium business rapid incubation online; third, using Docker technology, Some of the buildup in building a continuous integration environment.
As a "spoiler", containerized technology is bound to face the challenge of adapting to the existing architecture of the company, this article will introduce you to 360 how to let Docker landed. Mainly includes three aspects: first, combining the company's business characteristics, how to make Docker adapt to the existing technology structure, complete the online environment rapid deployment capacity expansion; second, "make the product less expensive", use Docker to build PAAs environment to accelerate small and medium business rapid incubation online; third, using Docker technology, Some of the buildup in building a continuous integration environment.
Docker-based containerized technology is now a breeze, and everyone feels it may subvert the entire it landscape. When we first came into contact with Docker, we felt it was very good and there were many advantages that attracted us. Because of its subversive nature, we call it "spoiler."
Transform "spoiler" Docker
Let's start by looking at the pros of this spoiler:
- Namespace, cgroups virtualization, compared to traditional virtualization will have better performance, reflected in the production environment is a greater degree of utilization of resources.
- The startup speed is fast, and the virtual machine has to be 30 seconds-1 minutes faster, and its start-up creation is second-level.
- Mirrored layering technology solves the problem of rapidly changing the environment.
These advantages are fascinating to us and we very much want to use it in our production environment, but we find the ideal is beautiful and the reality is very cruel. Our previous infrastructure was the use of traditional virtual machine technology as virtual machines. These are some of the issues we will face when we use Docker:
- Can't ssh, how to troubleshoot emergency issues?
- How to monitor?
- How do basic services dock?
- The most important question: Is this thing stable, and online business is certainly not going to go wrong.
So, when we applied Docker, we were hesitant because, in the way it was recommended, we couldn't immediately use the business online. Because Docker itself has certain requirements for the architectural design of the business, for example, we often say that the container is stateless, do not leave intermediate data in the container. We found that the company's business structure was difficult to transform and involved everything, so we decided to make Docker adapt to the company's architecture.
Next we are going to solve the problem of Docker technology "landing".
Our main points for Docker transformation are:
- The container is internally bound to an independent IP.
- Open the multi-process service inside the container.
- Automatically add monitoring.
- CPU quota hard limit.
- The container binds the standalone IP so that the external can ssh directly.
We consider running multiple process services inside the container, because the default container only opens a process, which does not meet our requirements, so we made a bold transformation. We even realized it in the mirror.
chkconfigMake the previous RPM package natively available.
Auto-add monitoring lets you create containers that are automatically added to Zabbix. CPU quotas Hard Limit the Docker 1.7 version is already supported, and we've implemented it ourselves before.
After retrofitting Docker to support these functions, we have developed a set of dispatching system, which is responsible for how to create containers on the cluster, we also investigate some open-source scheduling systems, find that they do not meet the requirements, so they developed a set.
Through these means we can let Docker technology "landed", and the benefit is that the previous system we want to go online new business takes about 40 minutes, using Docker shortened to 5 minutes.
This is the first part of sharing because the "spoiler" Docker is in trouble, we've made some changes to it to better fit the company scene and let technology "hit the ground."
Make an internal PAAs platform based on Docker
Then we made an internal PAAs platform based on Docker. The company will go online a lot of business every day, some of these business is a large volume of important business, some are with a trial-and-error nature of small business.
Traditional business on-line steps will be very rigorous, the process will be long, these processes in fact, the stability of the business will be guaranteed. Some of the small business of the wrong nature, using the same process becomes less appropriate, so we want to accelerate the small business on-line process, so that they can quickly go online, to verify their own value. Based on this consideration, and the nature of Docker is particularly suitable for this.
This is one of the interfaces, primarily the front-end Web UI to access a scheduling layer, and the dispatch layer creates containers by calling the Docker API. Currently, the PAAs platform supports languages such as PHP, node. js, Python, and Java.
In addition to creating containers, we also need to create a git repository, configure access agents, and so on, in short, the development of a key to the business to go online, as long as he passed the code can be online.
At present, the platform ran the 300+ business, so that a lot of research and development as long as there is an idea, you can quickly implement on-line, very popular with them.
This is also the second part of our application Docker, which accelerates business incubation through a private PAAs platform.
About Continuous integration
The third part is about continuous integration.
Continuous integration is, of course, the purest form of Docker, with "dockerfile-building mirrors-creating new containers" to make changes to the environment.
This piece is more complex, we have roughly divided 9 modules, such as dispatch module, monitoring module, storage module and so on.
First we made a configuration transformation module to convert the Dockerfile, so that it can unify the image building standards, but also reduce the cost of writing dockerfile learning.
The dispatch module directly uses the Mesos and Marathon, mirroring Registry directly using Registry V2 because it performs better for high concurrency support, and finally mirroring the building block, using Jenkins CI.
But we found a problem: mirroring is not really fast in high concurrency. For example, with a RPM package, SSH will certainly be much faster than a rebuild. So we did a lot of optimization in the mirror build this piece, now the result is 100 tasks simultaneously build we can also achieve the same efficiency as traditional cluster management like puppet.
Q&a
Q: How long has it taken to develop my own scheduling system, and have you encountered special technical difficulties?
A: About 2 engineers one months, not too many difficulties. Because the dispatch logic is relatively simple.
Q: You just said that SSH can be used directly by binding a separate IP.
A: By binding a separate IP can directly use SSH, the official about the network of the document has introduced the implementation of the scheme.
Q: The General Docker Service package is no daemon, then if you restart the service, the container will also exit, how to debug?
A: You can use SUPVIRSOD or monit to encapsulate no daemon.
Q: What is the registration of your service found?
A: Our infrastructure group has developed a name called Qconf, which has been open source on GitHub.
Q: You said that Zabbix do a container monitoring, that there is a host-based monitoring mode? Because, as far as I know, you have to run a proxy for each container.
A: We are using the host to install the Zabbix agent, through Zabbix self-discovery to dynamically obtain a list of containers, and then based on a custom monitoring script to obtain the monitoring value of each container.
Q: Are your businesses running in Docker?
A: At present 360 of a lot of Web2.0 business has been running on the above, such as film and television, news, free wifi and so on.
Q: Docker recommends no status, so does it mean that data is not recommended? such as MySQL, or through-V to solve?
Answer: This is actually a data storage problem, you can use distributed storage to store data, as long as the data and the logical separation of the container is stateless.
Q: Docker recommends no status, so does it mean that data is not recommended? such as MySQL, or through-V to solve?
A: I understand that the container is stateless and is created based on the image and can be used online immediately.
Q: How stable is the online docker?
A: The current operation is very stable, there is no abnormal collapse of the container and so on.
Q: Container run multiple processes, then the PID 1 process What control do you have, directly by the corresponding application or what else?
A: Before using Supervisord now use S6.
Q: Registry in the face of a lot of concurrency, have you tested the approximate performance ratio, the entire registry is mirror or other architectures?
A: Registry we are currently updating to V2, we test that the V2 performs very well on high concurrent pull and push, and that the mirrored storage uses shared storage so that registry can scale out horizontally.
Q: If the container configures IP that the user can access directly, can it be implemented in the host virtual machine based on the open vswitch, or will it be too dependent on the virtual machine network?
A: This can, in fact, we have also tested no problem, at that time based on stability considerations are not used.
Q: How is Qihoo's CPU quota management implemented?
A: This docker 1.7 has been implemented, and we are consistent with the official approach.
Q: How does a data store in a container work, and if it is shared storage how does it correspond?
Answer: You can try Glusterfs or ceph.
Q: The container is bound IP, after the container restarts IP to re-bind it, IP will change?
A: You need to re-bind to make an automated script.
===========================
The above content is organized according to the August 18, 2015 Night Group sharing content. Share people
Hussian (siliang5625633), joined Qihoo 360 in 2011, is engaged in the first-line operation and maintenance, and has extensive experience in the design of business operations and architecture online. Participates in the design and implementation of the Hulk private Cloud Platform architecture, leading the design and development of the internal PAAs platform. Dockone Weekly will organize the technology to share, welcome interested students add: LIYINGJIESX, into group participation, you want to listen to the topic can give us a message.
