This is a creation in Article, where the information may have evolved or changed.
"Editor's words" This time mainly talk about Docker's network solution, first of all, the existing container network scheme introduced, then focus on the characteristics of calico and technical points, as extended and compared to introduce the characteristics of Contiv, and finally give a comparison test results.
With the hot development of containers, more and more customers of the cloud have become more and more demanding on the characteristics of container network, such as:
- One container one IP;
- Multi-host container interconnection;
- Network isolation;
- ACL;
- Docking SDN and so on.
This time mainly to talk about Docker network solution, first of all, the existing container network solution, the next focus on calico characteristics and technical points, as extended and compared to introduce the characteristics of Contiv, and finally give a comparison test results.
Existing major Docker network scenarios
First of all, a brief introduction of the existing container network solutions, the Internet also looked at a lot of contrast, we have been based on the implementation of the way to divide, the following two solutions from the author Pengjove in Dockone share:
Tunnel Solutions
Through a tunnel, or overlay networking way:
- WEAVE,UDP broadcast, the machine set up a new BR, through PCAP interoperability.
- Open VSwitch (OVS), based on the Vxlan and GRE protocols, but the performance loss is more serious.
- FLANNEL,UDP broadcast, VxLan.
Tunnel scheme in the IAAS layer of the network application is also more, we agree that with the increase in the size of the node will increase the complexity, and the network problems to track down more trouble, large-scale cluster situation this is a point to consider.
Routing Scenarios
There is another type of approach that is implemented by routing, and the typical representative is:
- Calico, a BGP protocol-based routing scheme that supports very granular ACL control, has a high affinity to the hybrid cloud.
- Macvlan, from the logical and kernel layer of isolation and performance of the best solution, based on two layer isolation, so requires two layer of router support, most of the cloud service providers are not supported, so hybrid cloud is more difficult to implement.
Routing scheme is generally from 3 layer or 2 layer to achieve isolation and cross-host container interoperability, out of the problem is also easy to troubleshoot.
I think Docker 1.9 after the discussion of the container network solution, not only to see the implementation of the way, but also to see the network model of the "queued", such as whether you want to use Docker native "CNM", or CoreOS, Google's main push of the "company".
Docker libnetwork Container Network Model (CNM) Camp
- Docker Swarm Overlay
- Macvlan & IP Network drivers
- Calico
- Contiv (from Cisco)
The advantage of Docker Libnetwork is that it is native and tightly coupled with the Docker container life cycle, and the drawbacks can be understood as being native and being "kidnapped" by Docker.
Container Network Interface (MLM) Camp
- Kubernetes
- Weave
- Macvlan
- Flannel
- Calico
- Contiv
- Mesos
The advantage of the network is that it is compatible with other container technologies (e.g. Rkt) and the upper-level orchestration System (Kuberneres & Mesos), and the community is actively gaining momentum, kubernetes plus coreos, and the disadvantage is non-Docker native.
And from the above can also be seen, there are some third-party network solutions are "both camps", I personally think that the present state is a legitimate thing, but in the long run there is a risk, or be eliminated, or be acquired.
Calico
The next step is to introduce Calico, because it plays a more important role in both the CNM and the MLM camps. That is, has good performance, provides excellent isolation, but also has a good ACL control ability.
Calico is a pure 3-tier data center network solution and seamlessly integrates with an IaaS cloud architecture like OpenStack to provide controlled IP communication between VMS, containers, and bare metal.
By compressing the entire Internet's extensible IP Network principles to the data center level, Calico uses Linux kernel to implement an efficient vrouter for data forwarding at every compute node. Each vrouter, through the BGP protocol, is responsible for routing information that runs on its own workload as the entire Calico network-a small-scale deployment can be directly interconnected and can be done at scale with the specified BGP route reflector.
This ensures that the data traffic between all the workload ends up being interconnected via IP routing.
Calico node networking can take advantage of the network structure of the data center (either L2 or L3) without the need for additional NAT, tunneling, or overlay networks.
As shown, this ensures that the scheme is simple and controllable, and there is no packet unpacking, saving CPU computing resources while improving the performance of the whole network.
In addition, the Calico-based iptables also provides a rich and flexible network policy that ensures workload multi-tenant isolation, security groups, and other accessibility restrictions through ACLs on each node.
Calico Architecture
With this diagram above, let's go through the core components of calico:
- Felix,calico Agent, running on each node that needs to run workload, is mainly responsible for configuring the routing and ACLs information to ensure the endpoint connected state;
- ETCD, distributed key-value storage, mainly responsible for network metadata consistency, to ensure the accuracy of calico network status;
- The BGP Client (BIRD), which is responsible for distributing the kernel information of Felix to the current Calico network, ensures the validity of the communication between workload.
- BGP Route Reflector (BIRD), used in large-scale deployments, abandons the mesh mode of all nodes interconnection, and accomplishes centralized routing distribution through one or more BGP route Reflector;
Calico Docker Network Core Concept
From here we will "queued" CNM, through calico Docker libnetwork plugin way to experience and discuss Calico container network scenarios.
First look at the CNM model:
As you can see, CNM is based on 3 main concepts:
- Sandbox, contains the configuration of the container network stack, including interface, routing table and DNS configuration, corresponding implementation such as: Linux network Namespace; a sandbox can contain multiple network;
- Endpoint, as the sandbox access network media, the corresponding implementation such as: Veth pair, TAP; a Endpoint can only belong to a network, but also belong to a sandbox;
- Network, a group of endpoints that can communicate with each other, the corresponding implementation such as: Linux Bridge, Vlan;network has a large number of endpoint resources;
In addition, CNM needs to rely on two other key objects to complete Docker's network management functions, namely:
- Networkcontroller, Apis,docker Libnetwork, which provides distribution and management networks, supports multiple active networks Driver,networkcontroller allowing specific driver to be bound to a specified network;
- Driver, the network driver is not directly interactive to the user, it provides the realization of the final network function through plug-in access; Driver (including ipam) is responsible for managing a network, including resource allocation and recycling.
With these key concepts and objects, together with the Docker lifecycle, the ability to manage the container network through APIs can be accomplished, with specific steps and implementation details not discussed here, interested in the github:https://github.com/docker/ Libn. gn.md.
Next, we introduce the concept of two calico:
- Pool, which defines the range of IP resources available for the Docker network, such as: 10.0.0.0/8 or 192.168.0.0/16;
- Profile, which defines the set of Docker Network policy, consists of tags and rules, and each profile has a tag with the same name as the profile, and each profile can have multiple tags, saved in list form.
Profile Sample:
Inbound Rules:
1 Allow from Tag WEB
2 Allow TCP to ports 80,443
Outbound Rules:
1 Allow
Demo
Based on the above architecture and core concepts, we use a simple example to intuitively feel the Calico network management.
Calico to test for the purpose of cluster construction, the steps are very simple, here does not show, we can directly refer to Github:https://github.com/projectcali ... Me.md.
By default, there is a cluster of calico networks, the IP is: 192.168.99.102 and 192.168.99.103.
Calicoctl Status:
At the same time, there are already two IP pool created, namely: 10.0.0.0/26 and 192.168.0.0/16.
Calicoctl Pool Show:
The current cluster has also created different Docker Network by using Calico Driver and Ipam, and this demo only needs to use DataMan.
Docker Network ls:
Calicoctl Profile Show:
Below we use the DataMan network to start a container on each of the two slave machines:
Marathon json file for several people cloud-issued containers:
{"id": "/nginx-calico",
"cmd": null,
"CPUs": 0.1,
"Mem": 64,
"Disk": 0,
"Instances": 2,
"Container": {
"Type": "DOCKER",
"Volumes": [],
"Docker": {
"Image": "Nginx",
"Network": "HOST",
"Privileged": false,
"Parameters": [
{
"Key": "NET",
"Value": "DataMan"
}
],
"Forcepullimage": false
}
},
"Portdefinitions": [
{
"Port": 10000,
"Protocol": "TCP",
"Labels": {}
}
]
}
Two slave container IPs:
As can be seen, two slave on the container IP is: Slave 10.0.0.48, Slave2 192.168.115.193.
Slave Container Connectivity test:
IP Routing Implementation
Based on the above-mentioned Calico data plane concept diagram, in conjunction with our example, let's look at how Calico implements cross-host interoperability:
Two slave route:
In contrast to the two slave routing tables, we knew that if the container on slave 1 (10.0.0.48) wanted to send data to a container on slave 2 (192.168.115.193), it would match to the last routing rule, and the packet would be forwarded to slave 2 (192.168.99.103), the entire data flow is:
container, Kernel (cali2f0e) slave 1, one or more hops, (192.168.99.103) Slave 2, kernel (ca LI7D73) Container
In this way, the cross-host capacity period communication is established, and the entire data flow without NAT, tunneling, not involving packets.
Security Policy ACL
Calico's ACLs profile is primarily built on iptables and ipset, providing a rule definition that can be defined for each container level.
The specific implementation we can see the corresponding chain and filter rules through the iptables command, here we will not open the discussion.
Contiv
Http://contiv.github.io
Contiv is Cisco Open source for the container infrastructure, the main function is to provide policy-based network and storage management, is a new scaffolding for micro-services.
Contiv is able to integrate with the mainstream container orchestration system, including Docker Swarm, Kubernetes, Mesos and Nomad.
As shown, Contiv's "tempting" point is that its network management capabilities include both L2 (VLAN), L3 (BGP), Overlay (VxLAN), and the ability to connect with Cisco's own SDN product ACI. It can be said that it is possible to ignore the underlying network infrastructure and provide a consistent virtual network to the upper tier containers.
Contiv Netplugin Characteristics
- Multi-tenant network mixed on the same host;
- Integration of existing SDN solutions;
- Be able to collaborate with non-container environments and not rely on physical network specifics;
- Immediate effective container network Policy/acl/qos rules.
Network Scheme Performance comparison test
Finally, with the results of our simple performance test using Qperf, we chose VM-TO-VM, host, CALICO-BGP, CALICO-IPIP, and swarm overlay for comparison.
In fact, Contiv also tested, but because the Contiv test environment and other scenarios use different, do not have direct comparability, so not into the contrast chart. The intuitive feeling is that the OvS-based solution is not ideal, and there is time to continue the test comparison with the same environment, including the Contiv L3/L2 scheme and the introduction of Macvlan, flannel, etc.
Test environment: VirtualBox vms,os:centos 7.2,kernel 3.10,2 vcpu,2g Mem.
The bandwidth comparison results are as follows:
The delay comparison results are as follows:
Qperf command:
# Server-side
$ qperf
# Client Side
# Continuous 10s sends 64k packets, TCP_BW represents bandwidth, Tcp_lat indicates latency
$ qperf-m 64k-t 192.168.2.10 TCP_BW Tcp_lat
# starting from 1 exponential increment packet size
$ qperf-oo msg_size:1:64k:*2 192.168.2.10 TCP_BW Tcp_lat
Summarize
With the landing of the container, the network solution will become a "military" battleground, our number of people cloud is a lightweight PaaS container cluster Management cloud platform, in the choice of network options are:
- Performance, Calico and Macvlan all have a good performance, Contiv L3 (BGP) Theoretical performance is not bad;
- Universality, the basic requirement of calico is the IP layer can reach; Macvlan is not suitable for public cloud; Contiv compatibility with the underlying may be better;
- Programmability, to ensure that the entire network management process can be programmed, API, so that convenient integration into the product, do not need manual operation, calico and contiv have corresponding modules;
- The future development, this is what I say "queued", Docker CNM and CoreOS, kubernetes of the MLM, now also see unclear, Calico and Contiv are supported;
In summary, personal recommendation of attention and try to calico or Contiv as a container network solutions, there are problems or harvest, welcome to exchange and share at any time.
Q&a
Q: From the marathon JSON file you sent, did you expand the network on the bottom of Mesos? Does the container network pass through "parameters"?
A: It is the use of Marathon supported Docker parameters issued, equivalent to Docker run-net DataMan xxx.
Q:felix Configure routing based on the configured pool address segment, BGP client advertises the route? How is BGP's neighbor relationship built? Depends on the information in the ETCD?
A: Yes, Felix configures local kernel routing, BGP client is responsible for distributing routing information, BGP small-scale deployment takes the mesh approach established, all nodes interconnect n^2, large-scale deployments recommend deploying one or more BGP route reflector, Specifically responsible for routing information distribution.
Q: How is the information updated in DataMan this network? We know that BGP can find its own convergence network, so how is it designed at the network control level?
A: With the addition or deletion of the container using Datman, Felix is responsible for updating the local route, the BGP client is responsible for distributing outward, and BGP for Calico BGP and WAN is different, it just uses private as NUM, relative autonomy, The control level of the network can be programmed.
Q: How can calico solve the problem of address conflicts in multi-tenancy?
A: When a multi-tenant container is mixed on the same host, the network used is best not to use the same pool, so there is no conflict.
Q: If there are many containers in the cluster, then the rules of the corresponding routing table will increase, so will it affect the performance of the network?
A: Network performance will not have much impact, because the container multi-traffic multi-network pressure is large, it will increase the system load, after all, to configure the routing rules, synchronization network information; This part of us is also very concerned, but there is no specific test results.
Q: And is the routing information stored in the ETCD? How to ensure the reading performance of the routing table and maintain the information update of the routing table?
A: The effective route is certainly local, endpoint and other information is in the ETCD, these two problems are calico itself to solve the problem, which we are very concerned about, but there is no specific test results;
Q:calico The case of hop across multiple routes, does intermediate routing need to learn so many container routing tables?
A: Not required, the intermediate process is dependent on the original network between the nodes, as long as two nodes interoperability, all hop do not need to know calico internal container routing information.
Q: There is also a problem, that is, network management problems. At present, from the device manufacturer's website, there are several support Vxlan controller. Can these controllers work with the container OvS? At present, it is more difficult for us to buy foreign products here.
A: This is definitely a problem, I think as a platform side, we can do very little, if we want to change, it must be to the SDN equipment manufacturers to exert force, like the home will go for OpenStack adapter equipment to provide driver, However, Docker or container orchestration systems are required to achieve the industry standard level like OpenStack.
Q:calico can and flannel integration, CoreOS on the Kubelet also appeared calico and flannel?
A:calico in Dockercon 2016 should be a high-profile hug flannel, two integrated out a project called Canal, interested can go look for.
Q: I would like to ask the teacher, your Vxlan tag is calico, right?
A:calico no Vxlan,calico tag will correspond to Ipset,iptables used to match the filter rule.
The above content is organized according to the June 30, 2016 night group sharing content. Share people
Shangxiangyang, a few people cloud development engineer. Exposure to open source and OpenStack was earlier and was responsible for the development and maintenance of Red Hat's internal toolchain in Red Hat PnT (formerly HSS). is responsible for several people cloud research and development work, to Docker, Mesos research, familiar with and love cloud computing, distributed, SDN and other fields related technology. Dockone Weekly will organize the technology to share, welcome interested students add: Liyingjiesz, into group participation, you want to listen to the topic or want to share the topic can give us a message.
