This is a creation in Article, where the information may have evolved or changed.
"Editor's note" The Swarm project is one of the three Musketeers that Docker has launched to provide container trunking services to better help users manage multiple Docker engine users, using container cluster services like Docker engine. This sharing of content from the Swarm project status, swarm community status and swarm some of the future planning of the three aspects of swarm, the purpose is to let everyone have a complete understanding of swarm, and hope that more people use to swarm project.
Swarm background
There may be many applications in the real world, and the application itself can be complex, and the resources that a single Docker engine can provide may not be able to meet the requirements. And the application itself will have the reliability requirements, hope to avoid a single point of failure, so it is bound to be distributed in multiple Docker Engine. In this context, the Docker community has created a swarm project.
What is swarm?
Swarm the name of the project is particularly pertinent. In the wiki's interpretation, Swarm behavior refers to the animal's cluster behavior. For example, our common colony, the fish, the autumn to fly south of the flock can be called swarm behavior.
The Swarm project is just like this, by bringing together multiple Docker engine to form a large docker-engine that provides the container's cluster services externally. At the same time, the cluster provides swarm APIs that allow users to use Docker clusters as if they were using Docker engine.
Swarm Features
- External to the Docker API interface, the benefit is that if the existing system uses Docker engine, you can smooth the Docker engine cut to swarm, without altering the existing system.
- Swarm for users, previous experience with Docker can be inherited. Very easy to get started, learning costs and two development costs are relatively low. At the same time, Swarm itself focuses on Docker cluster management, which is very lightweight and consumes very little resources.
* "Batteries included but swappable", simply said, is the plug-in mechanism, the swarm of the various modules are abstracted out of the API, can be customized according to their own characteristics of implementation.
- Swarm's own support for Docker command parameters is relatively complete, and Swarm is currently being released synchronously with Docker. The new functionality of Docker will be reflected in swarm in the first time.
Swarm frame Structure
- Swarm provides two APIs, one of which is the Docker API for lifecycle management of container mirroring, and the other is the Swarm Cluster Management CLI for cluster management.
- Scheduler module, the main implementation of scheduling functions. When creating a container through Swarm, the Scheduler module selects an optimal node, which contains two sub-modules, namely filter and strategy, which filter the nodes to find the nodes satisfying the conditions (such as sufficient resources, normal nodes, etc.). Strategy is used to select an optimal node in a filtered node (e.g., to compare the nodes found, to find the nodes with the most resources) and, of course, filter/strategy users can customize them.
- Swarm to the cluster abstraction, abstracted cluster api,swarm support two kinds of clusters, one is swarm own cluster, and another one based on Mesos cluster.
- The leadership module is used for Swarm manager's own ha, which is implemented by the master and standby method.
- Discovery Service Discovery module, which is primarily used to provide node discovery capabilities.
- On each node, there will be an agent to connect the discovery service, escalate the IP port information of the Docker daemon, and Swarm Manager will read the node information directly from the Service Discovery module.
Swarm introduction of each module
Cluster Management
The Swarm Manager CLI is used for cluster management. You can look at this picture and create the cluster in three steps.
Once the Swarm container cluster is created, you can use Docker commands to create containers using the Swarm cluster like Docker engine.
Service discovery
Service discovery, which is used primarily for node discovery in Swarm, the agent on each node registers the IP port of Docker-egine with the service discovery system. The manager reads the node information from the Service Discovery module. Service discovery in Swarm supports the following 3 types of backend:
The first is the hosted Discovery service, which is the services discovery service provided by the Docker hub, which needs to be connected to extranet access.
The second, is KV distributed storage System, now supports ETCD, ZooKeeper, consul three kinds.
The third type is static IP. You can use a local file or specify a node IP directly, this way you do not need to start additional use of other components, generally used in debugging.
Scheduler
When the main user container of the dispatch module is created, select an optimal node. During the selection of the optimal node, there are two stages:
The first stage is filtering. Filter out the required nodes according to the conditions, the filter has the following 5 kinds:
- Constraints, constraint filter, can be based on the current operating system type, kernel version, storage type and other conditions to filter, of course, you can also customize the constraints, when starting daemon, through the label to specify the characteristics of the current host.
- Affnity, affinity filter, supports container affinity and mirroring affinity, such as a Web application, I want to put the DB container and the Web container together, it can be implemented through this filter.
- Dependency, depends on the filter. If a container is used when the container is created, the container that is created is --volume-from/--link/--net on the same node as the dependent container.
- The health filter, which is filtered according to the state of the node, removes the failed node.
- The Ports filter is filtered based on the port usage.
The second stage of scheduling is to select an optimal node based on the strategy. The following three strategies are available:
- Binpack, under the same conditions, select the node with the most resources, and through this strategy, the container can be gathered together.
- Spread, under the same conditions, select the node with the least resources, and through this strategy, the container can be distributed evenly on each node.
- Random, randomly select a node.
Leadership
The leadership module, which is primarily used to provide HA for Swarm manager itself.
In order to prevent Swarm Manager single point of failure, the introduction of the HA mechanism, Swarm Manager itself is stateless, so it is easy to implement ha. In the implementation process, the main standby mode, when the main node failure, will provide services from the new selection of the main process, the use of distributed lock implementation, now supports ETCD, ZooKeeper, consul three types of distributed storage, to provide distributed locks. When the standby node receives the message, it forwards the message to the master node.
The above is the framework of the various modules of the relevant introduction, down and everyone together to see, swarm and surrounding the integration of the project.
First look at the integration between the Three Musketeers.
Swarm integration with surrounding projects
The Three Musketeers were the three projects released by Docker at the end of last year, three of which could be closely coordinated. Take a look at this picture:
At the bottom is machine, which can be used to create a host with Docker-engine on different cloud platforms. Machine through the driver mechanism, currently supports the deployment of multiple platforms docker-egine environment, such as Amazon, OpenStack and so on. After the Docker engine was created, it was the swarm, swarm the docker-egnine on each host to provide container cluster services. Top of the Compose project, the compose project is primarily used to provide orchestration of container-based applications. The user describes an application consisting of multiple containers through a yml file, which is then parsed yml by compose, calling the Docker API to create a corresponding container on the swarm cluster.
We know that now around Docker has produced a great ecological circle. So swarm not only integrates with his own brothers, but also actively integrates with some of the surrounding projects. For example, Swarm is now ready to integrate with Mesos. When integrated with Mesos, Swarm is also integrated in the framework to achieve the interface required by the framework. This big feature is in the experiment stage.
The status quo of the swarm community
Swarm Project released at the end of last year, the development of a short period of six months, has reached the 0.4 version, is still in the rapid evolution stage. Swarm release cycle is currently released with Docker, basically two months a version, in the development process, the use of iterative development, basically every two weeks to complete a round of iterations. The approach to community participation is basically consistent with other communities. When you encounter problems, you can create issue in the community, and then describe the problem, preferably with environmental information and steps to reproduce the problem, which facilitates the positioning of the problem. You can also communicate directly via IRC or email. Swarm community is very welcome to participate, regardless of the problems encountered in the use of the bug, or swarm function is currently unable to meet everyone's place. Everyone is welcome to come forward and discuss together.
If you are interested in the code, you can refer to the Docker community's commit code process to submit the code, and you are welcome to participate in the swarm community to commit the code.
Swarm Future Planning
- The first is to support all Docker APIs, and now the support rate is around 95%, some of which are still problematic and need to be improved.
- The second piece is the network part, through the Libnetwork project, realizes overlay network.
- The third block is self healing, which can be implemented by this function, and when a node fails, the container on the failed node is created on another node.
- The four is global Scheduler. This feature is primarily used to create a container on each node. For example, to create a log container at each node for logging, this feature can be used.
- The last is volume, a community that has been discussing recently.
Q&a
q:kubernetes and Swarm compare to see how to choose?
A: A very open topic, according to the characteristics, choose the right for their own OK. Swarm external Docker API, self-light, learning costs, two times the cost of development is relatively low, itself is a plug-in framework. Functionally speaking, Swarm is a subset of q:kubernetes, personal feeling, compose+swarm =kubernetes.
Q:swarm What is the ultimate goal, just to manage the container, have you ever considered increasing the resource utilization, and will the resource elasticity be scaled up, eventually lifting all the machine load and preventing some low load or empty load from wasting resources?
a:auto-scaling ability, personal feeling behind may through compose to achieve, interested in words, can in swarm community to mention a proposal.
Q:swarm The selection of nodes can be customized, refers to the choice of strategy, feeling that only these three are not strong enough?
A: Yes, you can implement the corresponding API according to your own characteristics.
Q: How does calling the Swarm API and swarm Docker API security authentication work?
A: Security This part is through the SSL protocol to achieve communication security and authentication, support swarm external (such as with the client) between the provision of communication security, while the Swarm and Docker engine also support communication security.
How do q:swarm Cross-node link?
A: Cross-node is not currently supported, and if link is used, the container created and the link's container will be dispatched to the same node.
Q:swarm Scheduling system is also a plug-in form? Can I use Mesos resource scheduling?
The A:swarm Scheduler is a plug-in form. The Mesos uses a two-tier scheduling framework, the first layer, which is escalated by Mesos to the framework, the second layer, and the framework (swarm) 's own scheduler that allocates resources to the task.
How is the q:swarm IP managed? Are the various nodes under the Swarm dynamically assigned IP?
A: The current network part or use docker-engine own ability, follow-up and libnetwork integration, how to manage is being discussed.
does the Q:swarm support scheduling according to Docker tags?
A: Support, implemented by constraints filter.
Q: Is there any other plan or consideration for network integration besides libnetwork?
A:libnetwork itself also provides a plugin mechanism, a personal understanding, and good integration with other Web projects.
===========================
The above content is organized according to the September 8, 2015 Night Group sharing content. Share people
Line Super Bo, it cloud computing architecture and Design Department senior engineer, engaged in cloud computing direction of technical research, is currently mainly responsible for docker related technology in the field of cloud computing technology research and practice. Started to focus on the Docker swarm project in early 2015 and actively participated in Community contribution, becoming the first Docker community maintainer in China. Dockone Weekly will organize the technology to share, welcome interested students add: LIYINGJIESX, into group participation, you want to listen to the topic can give us a message.
