Kubernetes cluster pod uses host's local time zone setting

This is a creation in Article, where the information may have evolved or changed.

Kubernetes cluster has been set up, has been running very stable. Previous concerns have focused more on installation, configuration, and component debugging, and some of the minutiae has been overlooked. The settings for the time zone in the pod are one of them. I'm going to try to solve this problem today.

First, the problem phenomenon

On the master node of my kubernetes 1.3.7 Cluster, we execute:

# dateMon Feb 20 11:49:20 CST 2017

After that, randomly find the container in the pod on the node, and execute it through Docker exec in the container:

# docker exec -it 1975d68de07a /bin/bashroot@1975d68de07a:/# dateMon Feb 20 03:49:53 UTC 2017

We found that the current date of the output in Docker is inconsistent with the date output on the host. This does not seem to have much effect on the workings of the k8s cluster itself, at least for so long that it does not occur because of a different time setting than host. But for applications running in pods, this time-setting problem may cause a lot of trouble for the business.

In general, it's always right to keep the time settings in the pod consistent with the local times setting on the host. Here we will try to solve the problem.

Second, the Pod uses the host time zone setting scheme

I have two k8s clustered environments, one based on Ubuntu 14.04 node k8s 1.3.7 Environment, and one based on Ubuntu 16.04 node kubeadm k8s install 1.5.1 environment. Since Ubuntu 14.04 and Ubuntu 16.04 host have slightly different settings on the timezone, we also want to divide it into several cases (the OS of the Redhat system is not covered here, but the principle is the same):

0. Time zone settings on Ubuntu

On Ubuntu,/etc/localtime is the system's local time zone settings file that directly affects the current date output of the system. On Ubuntu 14.04 and Ubuntu 16.04, however, the content of this file is slightly different:

On Ubuntu 14.04,/etc/localtime is a regular file that stores the configuration data for the local time zone:

# file /etc/localtime/etc/localtime: timezone data, version 2, 2 gmt time flags, 2 std time flags, no leap seconds, 16 transition times, 2 abbreviation chars

On my node, the content is consistent with what/usr/share/zoneinfo/asia/shanghai points to, as if/etc/localtime were so derived:

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

On Ubuntu 16.04,/etc/localtime is a symbolic link that links to files:/usr/share/zoneinfo/asia/shanghai

# file  /etc/localtime/etc/localtime: symbolic link to /usr/share/zoneinfo/Asia/Shanghai

The real time zone settings file is stored under/usr/share/zoneinfo, and/usr/share/zoneinfo/asia/shanghai is also a symbolic link, pointing to/USR/SHARE/ZONEINFO/PRC:

# file /usr/share/zoneinfo/PRC/usr/share/zoneinfo/PRC: timezone data, version 2, 2 gmt time flags, 2 std time flags, no leap seconds, 16 transition times, 2 abbreviation chars

On 14.04 node, the contents of/etc/localtime and/USR/SHARE/ZONEINFO/PRC files are identical. However, in pod 14.04, the contents of these two files are different:

# docker exec -it fe936562b6ee /bin/bash# diff /etc/localtime /usr/share/zoneinfo/PRCBinary files /etc/localtime and /usr/share/zoneinfo/PRC differ

So, if you want the local time zone settings for the pod to be the same as the host, you have to do some "hands and feet" in the pod's manifest, and then we'll take a look at them in categories.

1. Host 14.04,pod 16.04

We run a 16.04 container at random on node 14.04, and we can see:

# docker run -it ubuntu:16.04 /bin/bashroot@bf7cec08df23:/# ls -l /etc/localtimelrwxrwxrwx 1 root root 27 Jan 19 16:33 /etc/localtime -> /usr/share/zoneinfo/Etc/UTC

The system time in the container is inconsistent with the host time.

Let's create a Docker image that uses Ubuntu 16.04:

//1604pod-image-dockerfileFROM ubuntu:16.04CMD ["tail", "-f", "/var/log/bootstrap.log"]

To build this image locally:

# docker build -f ./1604pod-image-dockerfile -t 1604podimage:latest .Sending build context to Docker daemon 5.632 kBStep 1 : FROM ubuntu:16.04 ---> f49eec89601eStep 2 : CMD tail -f /var/log/bootstrap.log ---> Using cache ---> 06ffb5c85d7cSuccessfully built 06ffb5c85d7c# docker images|grep 1604pod1604podimage                                                  latest              06ffb5c85d7c        28 minutes ago      129.5 MB

Let's write this manifest file that runs on the pod above 16.04:

//1604-pod-on-1404-host.yamlapiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: my-testpodspec:  replicas: 1  template:    metadata:      labels:        run: my-testpod    spec:      containers:      - name: my-testpod        image: 1604podimage:latest        imagePullPolicy: IfNotPresent        volumeMounts:        - name: tz-config          mountPath: /etc/localtime      volumes:      - name: tz-config        hostPath:          path: /usr/share/zoneinfo/Asia/Shanghai

We will mount the/usr/share/zoneinfo/asia/shanghai directly as the path/etc/locatime. Create the pod and check the system time in the pod:

# kubectl create -f 1604-pod-on-1404-host.yamldeployment "my-testpod" created# kubectl exec my-testpod-802169720-ehqlt dateMon Feb 20 14:19:13 CST 2017# dateMon Feb 20 14:19:15 CST 2017

You can see that the system time in the pod is consistent with the time on the host in the timezone.

2, Host 14.04, Pod 14.04

In Ubuntu 14.04, because the/etc/localtime itself stores the time zone setting, we need to mount it to the corresponding location in the pod. Our image demo is as follows:

//1404pod-image-dockerfileFROM ubuntu:14.04CMD ["tail", "-f", "/var/log/bootstrap.log"]

Build the Image:

# docker build -f ./1404pod-image-dockerfile -t 1404podimage:latest .Sending build context to Docker daemon 5.632 kBStep 1 : FROM ubuntu:14.04 ---> f2d8ce9fa988Step 2 : CMD tail -f /var/log/bootstrap.log ---> Running in 6815ca6fe9d9 ---> bc7f7de7690dRemoving intermediate container 6815ca6fe9d9Successfully built bc7f7de7690d# docker images|grep 1404pod1404podimage                                                  latest              bc7f7de7690d        8 seconds ago       187.9 MB

Pod manifest are as follows:

//1404-pod-on-1404-host.yamlapiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: my-testpodspec:  replicas: 1  template:    metadata:      labels:        run: my-testpod    spec:      containers:      - name: my-testpod        image: 1404podimage:latest        imagePullPolicy: IfNotPresent        volumeMounts:        - name: tz-config          mountPath: /etc/localtime      volumes:      - name: tz-config        hostPath:          path: /etc/localtime

As you can see, we attach the host's/etc/locatime to the/etc/localtime in the pod. After creating the pod, we look at the system time in the pod:

# kubectl exec my-testpod-2443385716-g9d4n dateMon Feb 20 14:44:57 CST 2017# dateMon Feb 20 14:44:59 CST 2017

As you can see, the two are already consistent on the time zone setting.

3. Host 16.04,pod 16.04

As a result of the above bedding, the following two cases, in view of the length, I will briefly describe. Here we will also take advantage of the two image:1404podimage:latest and 1604podimage:latest created above.

The pod's manifest file is as follows:

//1604-pod-on-1604-host.yamlapiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: my-testpodspec:  replicas: 1  template:    metadata:      labels:        run: my-testpod    spec:      containers:      - name: my-testpod        image: 1604podimage:latest        imagePullPolicy: IfNotPresent        volumeMounts:        - name: tz-config          mountPath: /etc/localtime      volumes:      - name: tz-config        hostPath:          path: /usr/share/zoneinfo/Asia/Shanghai

After you create the pod, look at the system time:

# kubectl exec my-testpod-3193072711-7kwdl dateMon Feb 20 14:55:00 CST 2017# dateMon Feb 20 14:55:31 CST 2017

The system time in the host and Pod is the same in the time zone.

4. Host 16.04,pod 14.04

The pod's manifest file is as follows:

//1404-pod-on-1604-host.yamlapiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: my-testpodspec:  replicas: 1  template:    metadata:      labels:        run: my-testpod    spec:      containers:      - name: my-testpod        image: 1404podimage:latest        imagePullPolicy: IfNotPresent        volumeMounts:        - name: tz-config          mountPath: /etc/localtime      volumes:      - name: tz-config        hostPath:          path: /usr/share/zoneinfo/Asia/Shanghai

Create the pod to compare the time in the pod with the host time:

# kubectl exec my-testpod-3024383045-xqbcv dateMon Feb 20 14:58:54 CST 2017# dateMon Feb 20 14:58:49 CST 2017

The system time in the host and Pod is the same in the time zone.

Third, summary

The above-mentioned manifest files and dockerfile file source can be downloaded here, you may need to make some changes according to your own k8s environment.

Weibo: @tonybai_cn
Public Number: Iamtonybai
Github.com Account: Https://github.com/bigwhite