Kubernetes Container Network Interface (midonet) design and implementation of the network plug-in. md

This is a creation in Article, where the information may have evolved or changed.

Overview of related principles

What is the first thing to say about the MLM?

The MLM (Container network interface) is an operation container network specification, including method specification, parameter specification and so on.
It only cares about the network connection of the container, allocates network resources when the container is created, and deletes the allocated resources when the container is deleted. Because of this focus, there is a wide range of support, and the specifications are easy to implement. The MLM interface only needs to implement two methods, one is called when the container is created, and one is called when the container is deleted.

Image

Kubernetes how to support and run a plug-in that complies with the MLM specification

Kubernetes first completes the network resource setting of the (Pod) container in the form of a plug-in. Built-in plugins include: Cni,kubenet,hostport and more. Here is a brief talk about Kubenet. This is a simple network plug-in that creates a br0 bridge on each machine, and sets the IP connection to the Br0 bridge for each pod according to PODCIDR. Sub-mode can be combined with some network routing tools to complete a small-scale cluster network pod interconnection. We mainly talk about the extension of the MLM. Kubernetes supports the MLM specification with the use of the MLM plug-in, and invokes various network plugins developed by other vendors and individuals to comply with the specifications of the company, such as Calico Flannel . K8s by default, port mapping is not supported by the MLM mode. K8s set the Container network to none, completely to the plug-in to manage the container network resources.

Image

What are the network resources mentioned many times?

Container network resources include: Virtual network card, IP address, DNS, network Routing and so on. The container uses a separate network namespace and can have its own network resource information. This information data is configured by different MLM plugins to the container according to the implementation of different SDN networks.

Midonet SDN Network

Midonet is a network virtualization software developed by the SDN company Midkura of Japan, which is based on the underlying physical facilities to achieve network virtualization, distributed, decentralized, multi-layered features, mainly as the default network components in OpenStack, can enable virtual network solutions, Specifically designed for network infrastructure, services for cloud platforms such as OpenStack, and virtualization of their network storage stacks. Midonet each tenant is assigned a logical router, the tenant and tenant are isolated from each other, the tenants can communicate with each other, midonet support L2 switching, L3 routing, L4 load balancing

Stateful and stateless NAT, logical and distributed firewalls, BGP with ECMP support. Its architecture mainly consists of the following components:

Midolman (midonet Agent): midonet

Agent installed in each compute node, responsible for establishing network traffic control and provide distributed Midonet network services, routing, NAT and so on he put the relevant virtual network information to NSDB.

Network State

Database (NSDB): Storage network configuration and status, network topology, routing, midonet non-centralized processing network functions, handled by midonet agent, midonet Agent will do real-time synchronization with Nsdbs when there are changes will be timely synchronization and update Nsdb
Midonet supports large-scale SDN clusters, and its architecture theoretically supports tens of thousands of nodes. We can use Midonet to complete the pod network interconnection within the k8s cluster.

MIDONET Network structure model under multi-tenancy

SDN (software Defined network), Midonet software defines the network components you know. The following is a brief introduction to several core software definition concepts:

• Router (Router)
  A tenant corresponds to a router, connected to the same router bridge network. Midonet will create a privierrouter, all tenants router connected to the Privierrouter and extranet. Equivalent to a router intranet interoperability, connect the superior router access to the public network.
• Bridge (Bridge)
  A tenant can have multiple bridges, each with a different network segment. For example, a bridge network segment 192.168.0.0/24 can have up to 253 virtual devices connected to this bridge.
• Port (device communication port)
  Between router and router, the communication interface between router and bridge.
• Route (Route)
  Routing rules, which define the rules for router traffic-packet forwarding ports.
• Rule (filter rules)
  Defines the package filter criteria. Similar to iptables.

Image

Midonet-based kubernetes of the implementation of the MLM plugin

Midonet data exchange works on layer three, but it does not provide IP address management (IPAM) itself, so the midonet-based MLM plugin needs to do the following: IPAM, tenant router, bridge creation, container NIC creation, And all end-to-end connection and routing filtering rules are created.

Ipam

Need to complete two levels of IP management, router level of address management, each router has an IP address, and the global unique does not conflict. Each bridge has a unique network segment, and the connected virtual network card has a globally unique IP.
Ipam has many implementations, and the MLM plugin is a stateless application, perhaps you need a daemon to complete ipam work. Based on the idea of simplifying the architecture, we use ETCD to store IP data and operate the ETCD directly from the plugin. Complete the use and release of the IP.

Tenant Network Initialization

When a new tenant creates a container for the first time, the initialization of the tenant virtual appliance is created, and we have described the virtual devices that a tenant needs to create, and here I'll tell you the details.
The midonet provides REST-API to operate the virtual appliance. Note here that different versions of the API are used according to the different versions of midonet used.
Https://github.com/barnettZQG/golang-midonetclient
Encapsulates the Golang version of the Midonet API operation method, which supports 1. and 5. API version.
The steps to create are as follows:

1. Create a tenant and invoke the Keystone API.
2. Create the router and include the Create and enter chain.
3. Create Privierrouter port and assign IP, create router port and assign IP. Create a Portlink connection to two ports.
4. Create a corresponding routing rule for the chain created earlier
5. Create a package filter rule for the port created earlier
6. Create a default bridge. and create a port connection on the router.
7. Stores the related data created above into ETCD.

Container network card creation and network binding

Virtual Ethernet Pair

Abbreviation Veth pair, is a pair of ports, all the packets from one end of the port will come out from the other side, and vice versa. The two ends can exist in different network spaces (networks Namespace). After the container has been created successfully with a network space, k8s calls the MLM Plugin Add method for network setup. The plug-in first creates a pair of veth pair. Place one end in the host network space and call the Midonet binding API to bind it to a port on bridge. The other end is inside the container and gives the IP address, depending on the bridge segment currently in use.
Same principle as the Docker0 NIC.

Set up intra-container routing rules

Set the default route to the network card created above. For example, the network card created above is named Eth0.

Set up DNS

Set up some DNS information as needed.

How does it work?

1. Use the shell command.

ip link *ip netns *ip address *ip route *

The above command detailed use method online many, here no longer describes.

1. Golang NetLink Library

Https://github.com/vishvananda/netlink
Defines the interface related to the user space associated with the Linux kernel for network cards.

Considerations for the implementation of the MLM plugin

1. The Add and delete operations of the MLM plugin should be idempotent, that is, the same parameters should have the same effect no matter how many times they are called.
2. The MLM plug-in should support concurrency, primarily the creation of tenant-related components and strong consistency in IP address assignment.
3. The MLM plug-in has certain specifications, please refer to: HTTPS://GITHUB.COM/CONTAINERNETWORKING/CNI

Good rain cloud midonet MLM Plugin Open source

Good clouds Open source Etcd version-based midonet the MLM plugin has all the features and features mentioned above.
Github:https://github.com/goodrain/midonet-cni