Kubernetes (k8s) container Runtime (CRI)

The bottom of the Kubernetes node is supported by a software called a "container runtime," which is responsible for things like starting and stopping containers. The most well-known container runtime is Docker, but it is not unique. In fact, this field has developed rapidly in the container runtime. To make the expansion of kubernetes easier, we have been polishing the k8s plug-in API that supports container runtime: the Container Runtime interface (Container runtime Interface, CRI).

What is CRI?

Each container runs with its own strengths, and many users want Kubernetes to support more runtimes. In the Kubernetes 1.5 release, we introduced cri– a plug-in interface that allows Kubelet to support multiple container runtimes without compiling. The CRI contains a set of protocol BUFFERS,GRPC APIs, associated libraries, and additional specifications and tools under active development. The CRI is currently an alpha version.

Support for replaceable container runtimes is not the first in the concept of kubernetes. In the 1.3 release, we introduced the Rktnetes project, which allows the Rkt container engine to be an alternative to the Docker container runtime. However, both Docker and RKT need to be integrated directly into the Kubelet source through an internal, less-stable interface. Such an integration process requires familiarity with the kubelet internal principles, and can also cause significant maintenance repercussions in the kubernetes community. All these factors are causing great difficulties in the early stages of container operation. We eliminate these barriers by providing a clearly defined abstraction layer where developers can focus on building their container runtime. This is a very small step, but it is very important for a truly pluggable container to run and to build a healthier ecosystem.

CRI Overview

When Kubelet communicates with the container runtime (or when the CRI plug-in fills the container runtime), the Kubelet is like a client, and the CRI plug-in is like the corresponding server. They can communicate between them through a UNIX socket or a GRPC framework.

The Protocol Buffers API contains two GRPC services: ImageService and Runtimeservice. ImageService provides RPC to pull, view, and remove mirrors from the mirror repository. Runtimeserivce includes RPC for Pods and container lifecycle management, and calls to interact with containers (Exec/attach/port-forward). A single-block container runtime is capable of managing mirrors and containers (for example, Docker and Rkt), and both services are available through the same socket. This socket can be set in Kubelet by identifying –container-runtime-endpoint and –image-service-endpoint.

Life Cycle Management

For Pod and container life cycle management, the CRI provides the following mechanism:

Service Runtimeservice {
Sandbox operations.
RPC Runpodsandbox (runpodsandboxrequest) returns (Runpodsandboxresponse) {}
RPC Stoppodsandbox (stoppodsandboxrequest) returns (Stoppodsandboxresponse) {}
RPC Removepodsandbox (removepodsandboxrequest) returns (Removepodsandboxresponse) {}
RPC Podsandboxstatus (podsandboxstatusrequest) returns (Podsandboxstatusresponse) {}
RPC Listpodsandbox (listpodsandboxrequest) returns (Listpodsandboxresponse) {}
Container operations.
RPC CreateContainer (createcontainerrequest) returns (Createcontainerresponse) {}
RPC Startcontainer (startcontainerrequest) returns (Startcontainerresponse) {}
RPC Stopcontainer (stopcontainerrequest) returns (Stopcontainerresponse) {}
RPC Removecontainer (removecontainerrequest) returns (Removecontainerresponse) {}
RPC Listcontainers (listcontainersrequest) returns (Listcontainersresponse) {}
RPC Containerstatus (containerstatusrequest) returns (Containerstatusresponse) {}
...
}

A pod is formed in a set of application containers in a resource-constrained isolated environment. In CRI, this environment is called Podsandbox. We deliberately leave some space for the containers to run with different podsandbox based on their internal principles. For hypervisor-based runtimes, Podsandbox may represent virtual machines. For others, such as Docker, it could be a Linux namespace. This podsandbox must follow the resource definition of pod. In the V1ALPHA1 version of the API, Kubelet will create a set of processes under the pod-level cgroup restriction and pass it to the container runtime, which is implemented.

Before the pod starts, Kubelet calls Runtimeservice.runpodsandbox to create the environment, including setting up the network for the pod (for example, assigning IP), and so on. When Podsandbox is started, separate containers can be created/started/stopped/removed separately. To remove Pod,kubelet, the Podsandbox is stopped and removed before all containers are stopped and removed.

Kubelet is responsible for managing container lifecycle through RPC, testing container lifecycle hooks and health/readability checks, and providing a restart strategy for pods.

Container-centric interface

Kubernetes has a declarative API for pod resources. We consider a possible design to enable the CRI to reuse this declarative pod object in its abstraction, giving the container runtime the freedom to implement and test the logic to achieve the desired state. This greatly simplifies the API and allows the CRI to be compatible with the broader runtime. We discussed this approach at an early stage of design, but it was rejected for several reasons. First, Kubelet has many pod-level features and specific techniques (such as Crash-loop backoff logic), which can be a huge burden for all runtime re-implementations. Second, it is increasingly important that the definition of pod is updated quickly. As long as Kubelet directly manages the container, many new features (such as init container) do not require any changes to the underlying container runtime. The CRI contains a necessary container-level interface so that the runtime can share these features and have a faster development speed. Of course this does not mean that we deviate from the "level triggered" philosophy. Kubelet is responsible for ensuring that the actual state changes to the desired state.

Interactive requests

Service Runtimeservice {
...
Execsync runs a command in a container synchronously.
RPC Execsync (execsyncrequest) returns (Execsyncresponse) {}//Exec prepares a streaming endpoint to execute a command in The container.
RPC Exec (execrequest) returns (Execresponse) {}//Attach prepares a streaming endpoint to Attach to a running container.
RPC Attach (attachrequest) returns (Attachresponse) {}//Portforward prepares a streaming endpoint to forward ports from a Podsandbox.
RPC Portforward (portforwardrequest) returns (Portforwardresponse) {}
...
}

Kubernetes provides features that allow users to interact with pods and their internal containers (for example, Kubectl exec/attach/port-forward). Kubelet now supports these features by calling the container's native methods or using the tools available on the node, such as Nsenter and Socat. Using these tools on a node is not a good way to migrate because most of these tools assume that pods are isolated through the Linux namespace. At CRI, we explicitly define these calls to allow specific runtime implementations.

Another potential problem is that Kubelet's implementation today is Kubelet handles all streaming connection requests. So this will bring bottlenecks to the network traffic of the nodes. In designing the CRI, we adopted this feedback to support the run-time guard against the middleman. The container runtime can start a separate streaming server that corresponds to the request (which may even be used for the Pod Audit resource) and return the address to Kubelet. Kubelet then returns this information to the Kubernetes API server, which opens a stream connection directly to the server provided by the runtime and communicates it to the client.

Kubernetes (k8s) container Runtime (CRI)