There is a detailed introduction about the official website of
Kubernetes certificate management, but it involves a lot of content and is slightly different under different versions. Many Chinese articles have information errors, such as kubelet server
certificate is no longer automatically approve certificate creation request due to security issues, manual operation is required.
Foreword
There are many
Kubernetes cluster components, and communication between components is encrypted using TLS certificates. In the operation and maintenance cluster, it is very important to do certificate configuration and update. This article introduces the certificates that a typical K8S cluster created by kubeadm has, and how to manage them.
Note: In this article, kubeadm uses version 1.13, and the certificate generation command before version 1.13 is still in alpha state. For clusters above 1.13, use the corresponding version of kubeadm. For versions below 1.13, you can use version 1.13 of kubeadm. The certificate itself will not change significantly due to the cluster version. It is also possible to use the equivalent openssl or cfssl commands.
Overview and management of Kubernetes cluster certificates
A Kubernetes cluster can be divided into two parts. One part is control-plane, mainly including etcd, kube-apiserver, kube-controller-manager, kube-scheduler. Generally deployed on 1 or 3 nodes. In multi-node deployment, the respective certificate creation methods are the same, no difference. The other part is worker nodes. Each node has a kubelet process that requires a certificate. The management method is the same, but the node hostname in CN is different. The following describes the certificates used by each component.
Environment variables
Different clusters can customize the certificate storage path and various cluster parameters. To make the sample commands common, define the following environment variables:
Note: The cluster parameters can also be passed through the kubeadm configuration file. The cluster deployed by kubeadm should manage the configuration file for future maintenance.
CERT_DIR
Certificate storage path, kubeadm defaults to /etc/kubernetes/pki.
SERVICE_CIDR
Cluster Service CIDR, the default is 10.96.0.0/12.
CLUSTER_DOMAIN
Cluster domain name, default is cluster.local
APISERVER_CERT_EXTRA_SANS
kube-apiserver certificate for additional SANS. The default is empty. If there is an LB or proxy in front of the apiserver, you need to add the LP, proxy IP or domain name, so that the client can correctly connect to apisever through the LB or proxy.
Control Plane
Quickly view the expiration time of all certificates on the control plane node
CERT_DIR=${CERT_DIR:-/etc/kubernetes/pki}
for i in $(find $CERT_DIR -name'*.crt' -o -name'*.pem'); do
echo $i
openssl x509 -enddate -in $i -noout
done
for f in $(ls /etc/kubernetes/{admin,controller-manager,scheduler}.conf); do
echo $f
kubectl --kubeconfig $f config view --raw -o jsonpath='{range .users[*]}{.user.client-certificate-data}{end}' | base64 -d | openssl x509 -enddate -noout
done
etcd
As the
database of kube-apiserver, etcd, its deployment may have customized deployment and management due to HA or other special requirements. For customized deployment, refer to official documents to generate and manage certificates. The following uses kubeadm deployment as an example:
Overview
$CERT_DIR/etcd/ca.{crt,key}
CA certificate and key of etcd
$CERT_DIR/etcd/healthcheck-client.{crt,key}
Client certificate and key for health check. kubeadm is configured in livenessProbe to detect etcd.
$CERT_DIR/etcd/server.{crt,key}
etcd server certificate and key
$CERT_DIR/etcd/peer.{crt,key}
peer certificate and key for etcd
Update
The subcommands in kubeadm have been used to update the corresponding certificates:
kubeadm init phase certs etcd-ca
kubeadm init phase certs etcd-healthcheck-client
kubeadm init phase certs etcd-peer
kubeadm init phase certs etcd-server
kube-apiserver
Overview
$CERT_DIR/apiserver.{crt,key}
server certificate, key
$CERT_DIR/apiserver-kubelet-client.{crt,key}
Client certificate and key used to connect to kubelet
$CERT_DIR/front-proxy-client.{crt,key}
Used to connect to other apiserver (aggregated apiserver) or webhook plugins and other client certificates and keys
$CERT_DIR/apiserver-etcd-client.{crt,key}
Client certificate and key used to connect etcd
Update
kubeadm init phase certs apiserver -v 4\
--apiserver-cert-extra-sans "$APISERVER_CERT_EXTRA_SANS" \
--service-cidr "$SERVICE_CIDR" \
--service-dns-domain "$CLUSTER_DOMAIN" \
--cert-dir "$CERT_DIR"
kubeadm init phase certs apiserver-kubelet-client -v 4 \
--cert-dir "$CERT_DIR"
kubeadm init phase certs front-proxy-client -v 4 \
--cert-dir "$CERT_DIR"
kubeadm init phase certs apiserver-etcd-client -v 4 \
--cert-dir "$CERT_DIR"
Note: "kubeadm init phase certs apiserver-etcd-client" assumes that etcd is deployed and managed by kubeadm. If etcd is a customized deployment, the kube-apiserver's etcd client certificate needs to be specially generated according to the etcd deployment.
kubeconfig configuration for kube-controller-manager, kube-scheduler, and admin
Overview
/etc/kubernetes/admin.conf
/etc/kubernetes/kube-controller-manager.conf
/etc/kubernetes/kube-scheduler.conf
The above are the kubeconfig configuration files for the admin user, kube-controller-manager, and kube-scheduler.
Update
kubeadm init phase kubeconfig admin \
--cert-dir $CERT_DIR
kubeadm init phase kubeconfig controller-manager \
--cert-dir $CERT_DIR
kubeadm init phase kubeconfig scheduler \
--cert-dir $CERT_DIR
Node
Only the kubelet component on the node needs to be configured with a certificate. If client/server certificate scrolling is not configured, manual processing is required, which is unrealistic in a large-scale cluster. This article only considers the processing method after configuring the certificate rolling. If not configured, it is recommended to open.
Open kubelet client/server certificate scrolling
kubelet configuration file
featureGates:
# 1.11 and earlier versions RotateKubeletServerCertificate is closed by default, you need to enable RotateKubeletServerCertificate in kubelet and kube-controller-manager.
RotateKubeletServerCertificate: true # false before v1.11
rotateCertificates: true
serverTLSBootstrap: true
kubelet command line
# 1.11 and earlier versions RotateKubeletServerCertificate is closed by default, you need to enable RotateKubeletServerCertificate in kubelet and kube-controller-manager.
--feature-gates RotateKubeletServerCertificate=true
--rotate-certificates
--rotate-server-certificates
Reboot bootstrap after kubelet certificate expires
If the existing cluster is not configured with certificate scrolling or unexpected conditions, the kubelet
certificate expires. In this case, the bootstrap certificate needs to be regenerated so that the kubelet can be restarted.
The default address of the bootstrap certificate is in /etc/kubernetes/bootstrap-kubelet.conf. Command line parameter configuration:
--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
kubeadm will use token as bootstrap token during initialization. If you need to restart bootstrap, you can use the following command to generate bootstrap configuration for the cluster nodes, and restart kubelet (note that you need to configure the client/server certificate to scroll).
for n in $(kubectl get nodes --no-headers | awk'{print $1}'); do
kubeadm init phase kubeconfig kubelet \
--cert-dir $CERT_DIR \
--node-name $n \
--kubeconfig-dir ./kubelets/$n
done
Note: Need to run on the control-plane node (in order to access the cluster ca certificate).
Update
After enabling the certificate scrolling, the kubelet client certificate will be automatically updated without maintenance. kubelet server certificate Because in the bare-metal environment, kubernetes cannot verify whether the node of the CSR submitted by kubelet is trusted, it needs manual approve CSR. Examples:
kubectl get csrs
kubectl certificate approve <csr>
Note: A simple program can be developed here, and an automatic approve can be made according to the actual cluster situation. Or after monitoring (see below), manual operation when it is about to expire.
Kubernetes cluster certificate monitoring
kube-apiserver's metrics
kube-apiserver will output the expiration information of its client certificate, if a client with expired certificate connects to apiser
ver can be known through its metrics.
apiserver_client_certificate_expiration_seconds_bucket
apiserver_client_certificate_expiration_seconds_bucket
kubelet's metrics
The client/server certificate manager of kubelet will output some metrics, which can be used for monitoring.
# HELP kubelet_certificate_manager_client_expiration_seconds Gauge of the lifetime of a certificate. The value is the date the certificate will expire in seconds since January 1, 1970 UTC.
# TYPE kubelet_certificate_manager_client_expiration_seconds gauge
kubelet_certificate_manager_client_expiration_seconds 1.56058818e+09
# HELP kubelet_certificate_manager_server_expiration_seconds Gauge of the lifetime of a certificate. The value is the date the certificate will expire in seconds since January 1, 1970 UTC.
# TYPE kubelet_certificate_manager_server_expiration_seconds gauge
kubelet_certificate_manager_server_expiration_seconds 1.56058932e+09
kube-apiserver, etcd server monitoring
The blackbox_exporter can be deployed for TLS certificate of kube-apiserver and etcd server (including any HTTPS server) for TLS detection and monitoring.
