Deep anatomy kubernetes API Server Trilogy-Part 2

In the previous section we discussed the API server overall, related terminology and request flow for requests. In this section of the article, we focus on exploring how to kubernetes the state of a link-descriptive object in a reliable, durable way to manage it. As mentioned in the previous article, API server itself is stateless, and it is the only component that can communicate directly with distributed storage ETCD.
Brief description of Etcd
In the *nix operating system, we typically use/etc to store the relevant configuration data. In fact, ETCD's name is the result of this development, after etc, add a "D" means "distributed" distributed. Any distributed system needs to have something like ETCD that can store system data so that it can retrieve relevant data in a consistent and reliable manner. In order to achieve distributed data access, ETCD uses the raft protocol. Conceptually, the data model supported by ETCD is a key-value (Key-value) store. In ETCD2, each key exists in a hierarchical structure, and in etcd3 this becomes a pervasive model, but it also preserves the compatibility of the hierarchical approach.

Using the containerized version of ETCD, we can create the tree above and retrieve it as follows:
$ docker Run--rm-d-P 2,379:2,379 \
--name TEST-ETCD3 QUAY.IO/COREOS/ETCD:V3.1.0/USR/LOCAL/BIN/ETCD \
--advertise-client-urls http://0.0.0.0:2379--listen-client-urls http://0.0.0.0:2379
$ Curl localhost:2379/v2/keys/foo-xput-d value= "some value"
$ Curl localhost:2379/v2/keys/bar/this-xput-d value=42
$ Curl localhost:2379/v2/keys/bar/that-xput-d Value=take
$ http Localhost:2379/v2/keys/?recursive=true
http/1.1 OK
content-length:327
Content-type:application/json
Date:tue, June 12:28:28 GMT
x-etcd-cluster-id:10e5e39849dab251
X-etcd-index:6
X-raft-index:7
X-raft-term:2
{
"Action": "Get",
"Node": {
"dir": true,
"Nodes": [
{
"Createdindex": 4,
"Key": "/foo",
"Modifiedindex": 4,
"Value": "Some value"
},
{
"Createdindex": 5,
"dir": true,
"Key": "/bar",
"Modifiedindex": 5,
"Nodes": [
{
"Createdindex": 5,
"Key": "/bar/this",
"Modifiedindex": 5,
"Value": "42"
},
{
"Createdindex": 6,
"Key": "/bar/that",
"Modifiedindex": 6,
"Value": "Take"
}
]
}
]
}
}
Now that we have a general idea of how ETCD works, we continue to discuss how ETCD is being used in kubernetes.
ETCD in a cluster
In Kubernetes, ETCD is an independent component of the face in the control plane. Before the Kubernetes1.5.2 version, we used the ETCD2 version, and after the Kubernetes1.5.2 version we turned to the ETCD3 version. It is worth noting that in the kubernetes1.5.x version ETCD is still using the V2 API model, which will then begin to become the V3 API model, including the data model used. This does not seem to have a direct impact on the developer's point of view, because API server is an abstract interaction with storage and does not care whether the backend storage implementation is ETCD v2 or v3. However, if you are standing in the perspective of a Cluster Administrator, you still need to know which version of ETCD is used, because the Cluster Administrator needs to perform some daily backup of the data and restore maintenance operations.
You can configure the API server related startup items in a way that uses ETCD, and the ETCD related startup key parameters for the API server are as follows:
$ kube-apiserver-h
...
--etcd-cafile string SSL Certificate Authority file used to secure ETCD communication.
--etcd-certfile string SSL Certification file used to secure ETCD communication.
--etcd-keyfile string SSL key file used to secure ETCD communication.
...
--etcd-quorum-read If True, enable quorum read.
--etcd-servers List of ETCD servers to connect with (Scheme://ip:port) ...
...
Kubernetes data stored in ETCD is stored in a JSON string or protocol buffers format. Let's take a look at a concrete example: Create a webserver pod in the Apiserver-sandbox namespace. Then we use the Etcdctl tool to view the relevant ETCD (ETCD version 3.1.0) data in this session.
$ cat Pod.yaml
Apiversion:v1
Kind:pod
Metadata
Name:webserver
Spec
Containers

• Name:nginx
  Image:tomaskral/nonroot-nginx
  Ports
  • Containerport:80

$ kubectl create-f Pod.yaml

$ etcdctl LS/
/kubernetes.io
/openshift.io

$ etcdctl Get/kubernetes.io/pods/apiserver-sandbox/webserver
{
"Kind": "Pod",
"Apiversion": "V1",
"Metadata": {
"Name": "Webserver",
...
Let's take a look at how this pod object is eventually stored in Etcd, via Kubectl create-f Pod.yaml. Depicts this overall process:

2. Clients (such as KUBECTL) provide an ideal state object, such as the V1 version in YAML format.
4. The KUBECTL converts YAML to JSON format and sends it.
6. For different versions of the same type of object, the API server performs a lossless conversion. Fields that do not exist in the old version are stored in annotations.
8. API server transforms the accepted object into a canonical storage version, which is specified by the API server and is generally the latest stable version, such as v1.
9. Finally, the object is parsed into a value via JSON or PROTOBUF, which is deposited into the ETCD by a specific key.
   We can determine the format in which you want to serialize data into ETCD by configuring the Kube-apiserver startup parameter--storage-media-type, which is application/vnd.kubernetes.protobuf format by default. We can also configure the--storage-versions startup parameters to determine the default version number of each group object that is deposited into the ETCD.
   Now let's look at how lossless conversions work, and we'll use the Kubernetes object horizontal Pod autoscaling (HPA) to enumerate the instructions. The HPA, as its name implies, controls the scaling of pods by monitoring the use of resources combined with Replicationcontroller.
   First we look for an API proxy (so that we can access it locally directly) and start Replicationcontroller, as well as the HPA.
   $ kubectl Proxy--port=8080 &
   $ kubectl create-f Https://raw.githubusercontent.com/mhausenblas/kbe/master/specs/rcs/rc.yaml
   Kubectl Autoscale RC Rcex--min=2--max=5--cpu-percent=80
   Kubectl Get Hpa/rcex-o Yaml
   Now you can use httpie--of course you can also use curl-to request a HPA object from the API server using the current stable version (AUTOSCALING/V1), or use the previous version (EXTENSIONS/V1BETA1), The differences between the two versions obtained are as follows:
   $ http Localhost:8080/apis/extensions/v1beta1/namespaces/api-server-deepdive/horizontalpodautoscalers/rcex > Hpa-v1beta1.json
   $ http Localhost:8080/apis/autoscaling/v1/namespaces/api-server-deepdive/horizontalpodautoscalers/rcex > Hpa-v1.json
   $ diff-u Hpa-v1beta1.json Hpa-v1.json
   {
   "Kind": "Horizontalpodautoscaler",
   • "Apiversion": "Extensions/v1beta1",
   • "Apiversion": "Autoscaling/v1",
     "Metadata": {
     "Name": "Rcex",
     "Namespace": "Api-server-deepdive",
   • "Selflink": "/apis/extensions/v1beta1/namespaces/api-server-deepdive/horizontalpodautoscalers/rcex",
   • "Selflink": "/apis/autoscaling/v1/namespaces/api-server-deepdive/horizontalpodautoscalers/rcex",
     "UID": "Ad7efe42-50ed-11e7-9882-5254009543f6",
     "Resourceversion": "267762",
     "Creationtimestamp": "2017-06-14t10:39:00z"
     },
     "Spec": {
   • "Scaleref": {
   • "Scaletargetref": {
     "Kind": "Replicationcontroller",
     "Name": "Rcex",
   • "Apiversion": "V1",
   • "Subresource": "Scale"
   • "Apiversion": "V1"
     },
     "Minreplicas": 2,
     "Maxreplicas": 5,
   • "Cpuutilization": {
   • "Targetpercentage": 80
   • }
   • "Targetcpuutilizationpercentage": 80
     We can see that the version of Horizontalpodautoscale has changed from V1beta1 to V1. API server is capable of lossless conversions before different versions, regardless of which version is actually stored in the ETCD.
     After understanding the entire storage process, let's look at how API server encodes the data, decoding it in JSON or protobuf in Etcd, and taking into account the ETCD version.
     API server saves all known Kubernetes object types in the Go Type Registry (registry) named scheme. In this registry, you define each type of kubernetes object and how to convert it, how to create a new object, and how to encode and decode the object as JSON or PROTOBUF.
     
     When API server receives an object from the client, such as KUBECTL, it can know the specific version number of the object through an HTTP path. First, an empty object is created for this object using the corresponding version scheme, and then the object content that is passed over by HTTP is decoded and converted via JSON or PROTOBUF. After decoding is completed, the object is created and stored in ETCD.
     There may be many versions in the API, which is often cumbersome to handle if you want to support direct conversions between each version. For example, there are three versions of an API, so it will support a direct conversion of one version to another two (such as v1?). V1ALPHA1, v1? V1beta1, v1beta1? V1ALPHA1). To avoid this problem, there is a special "internal" version in API server. When a conversion is required between two versions, it is converted to the internal version and then to the corresponding converted version. In this case, each version can be converted indirectly to any other version as long as the support can be converted to the internal version. So an object is first converted to the internal version, then converted to a stable version of V1, and then stored in ETCD.
     V1beta1? Internal? V1
     In the first step of the conversion, if some field users do not assign a value, these are assigned as a default value. For example, in v1beta1 there is definitely no new field in the V1 version. In this case, the user will definitely not be able to assign a value to this field in the V1beta1 version. At this point, in the first step of the conversion, we will assign a default value to this field to generate a valid internal.
     Calibration and access
     There are two important steps in the conversion process, as shown in:
     V1beta1? Internal? |? |? V1? Json/yaml? Etcd
     Admission validation
     Admittance and validation are the steps that must be passed before the object is created and updated to be stored in ETCD. Some of their rules are as follows:
11. Access (admission): see if some of the constraints in the cluster allow you to create or update this object and set some default values for the object based on the configuration of the cluster. There are many such constraints in kubernetes, here are some examples:
    Namespacelifecycle: If the namespace does not exist, all incoming requests under that namespace are rejected.
    Limitranger: Enforces restrictions on the utilization of resources in namespaces.
    ServiceAccount: Create service account for pod.
    Defaultstorageclass: If the user does not assign a value to Persistentvolumeclaims, set it to a default value.
    Resourcequota: Enforces quota constraints on the current user on the cluster and may deny requests if there are insufficient quotas.
13. Checksum (Validation): Checks whether the incoming object (during creation and update) is well-formed and whether the associated value is valid. Like what:
15. Check that the required fields are filled in.
17. Check that the string format is correct (such as lowercase only).
19. If there are some fields that are conflicting (for example, there are two containers with the same name).
    Validation (Validation) does not care about other types of object instances, in other words, it only cares about the static check of each object, regardless of the cluster configuration.
    Access (admission) can be started or disabled with flag--admission-control=<plugins>. Most of them can have a cluster management configuration. In addition, in Kubernetes 1.7, the webhook mechanism can be used to extend the access mechanism, using the controller to achieve the traditional verification of the object.
    Migrating Storage objects
    Last note on storage object migration: When Kubernetes needs to be upgraded to a new version, it is important to back up the data for the relevant cluster according to the relevant document steps for each version. This is due to the change of ETCD2 to ETCD3, on the other hand, due to the continuous development of the kind and version of the Kubernetes object.
    In Etcd, each object is present in the preferred storage version (preferred storage versions). However, over time, the objects in the ETCD store may exist in a very old version. If the object version is discarded at some time in the future, it will no longer be able to decode its protobuf or JSON. Therefore, you need to rewrite and migrate this data before the cluster upgrade. The following information can help with version switching:
    See the Cluster Management Documentation upgrade API version section. Upgrading to a different API version
    Next time, in the third part of the in-depth study of kubernetes Apiserver, we will discuss how to use custom Resource definitions to extend and customize API resources. Https://www.huaweicloud.com/product/cce.html

Deep anatomy kubernetes API Server Trilogy-Part 2