client-go的使用及源碼分析

本文個人部落格地址：http://www.huweihuang.com/article/source-analysis/client-go-source-analysis/ 1. client-go簡介 1.1 client-go說明

client-go是一個調用kubernetes叢集資來源物件API的用戶端，即通過client-go實現對kubernetes叢集中資來源物件（包括deployment、service、ingress、replicaSet、pod、namespace、node等）的增刪改查等操作。大部分對kubernetes進行前置API封裝的二次開發都通過client-go這個第三方包來實現。

client-go官方文檔：https://github.com/kubernetes/client-go 1.2 範例程式碼

git clone https://github.com/huweihuang/client-go.gitcd client-go#保證本地HOME目錄有配置kubernetes叢集的設定檔go run client-go.go

client-go.go

package mainimport (    "flag"    "fmt"    "os"    "path/filepath"    "time"    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"    "k8s.io/client-go/kubernetes"    "k8s.io/client-go/tools/clientcmd")func main() {    var kubeconfig *string    if home := homeDir(); home != "" {        kubeconfig = flag.String("kubeconfig", filepath.Join(home, ".kube", "config"), "(optional) absolute path to the kubeconfig file")    } else {        kubeconfig = flag.String("kubeconfig", "", "absolute path to the kubeconfig file")    }    flag.Parse()    // uses the current context in kubeconfig    config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)    if err != nil {        panic(err.Error())    }    // creates the clientset    clientset, err := kubernetes.NewForConfig(config)    if err != nil {        panic(err.Error())    }    for {        pods, err := clientset.CoreV1().Pods("").List(metav1.ListOptions{})        if err != nil {            panic(err.Error())        }        fmt.Printf("There are %d pods in the cluster\n", len(pods.Items))        time.Sleep(10 * time.Second)    }}func homeDir() string {    if h := os.Getenv("HOME"); h != "" {        return h    }    return os.Getenv("USERPROFILE") // windows}

1.3 運行結果

➜ go run client-go.goThere are 9 pods in the clusterThere are 7 pods in the clusterThere are 7 pods in the clusterThere are 7 pods in the clusterThere are 7 pods in the cluster

2. client-go源碼分析

client-go源碼：https://github.com/kubernetes/client-go

client-go源碼目錄結構 The kubernetes package contains the clientset to access Kubernetes API. The discovery package is used to discover APIs supported by a Kubernetes API server. The dynamic package contains a dynamic client that can perform generic operations on arbitrary Kubernetes API objects. The transport package is used to set up auth and start a connection. The tools/cache package is useful for writing controllers. 2.1 kubeconfig

kubeconfig = flag.String("kubeconfig", filepath.Join(home, ".kube", "config"), "(optional) absolute path to the kubeconfig file")

擷取kubernetes設定檔kubeconfig的絕對路徑。一般路徑為$HOME/.kube/config。該檔案主要用來配置本地串連的kubernetes叢集。

config內容如下：

apiVersion: v1clusters:- cluster:    server: http://<kube-master-ip>:8080  name: k8scontexts:- context:    cluster: k8s    namespace: default    user: ""  name: defaultcurrent-context: defaultkind: Configpreferences: {}users: []

2.2 rest.config

通過參數（master的url或者kubeconfig路徑）和BuildConfigFromFlags方法來擷取rest.Config對象，一般是通過參數kubeconfig的路徑。

config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)

BuildConfigFromFlags函數源碼

k8s.io/client-go/tools/clientcmd/client_config.go

// BuildConfigFromFlags is a helper function that builds configs from a master// url or a kubeconfig filepath. These are passed in as command line flags for cluster// components. Warnings should reflect this usage. If neither masterUrl or kubeconfigPath// are passed in we fallback to inClusterConfig. If inClusterConfig fails, we fallback// to the default config.func BuildConfigFromFlags(masterUrl, kubeconfigPath string) (*restclient.Config, error) {    if kubeconfigPath == "" && masterUrl == "" {        glog.Warningf("Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.")        kubeconfig, err := restclient.InClusterConfig()        if err == nil {            return kubeconfig, nil        }        glog.Warning("error creating inClusterConfig, falling back to default config: ", err)    }    return NewNonInteractiveDeferredLoadingClientConfig(        &ClientConfigLoadingRules{ExplicitPath: kubeconfigPath},        &ConfigOverrides{ClusterInfo: clientcmdapi.Cluster{Server: masterUrl}}).ClientConfig()}

2.3 clientset

通過*rest.Config參數和NewForConfig方法來擷取clientset對象，clientset是多個client的集合，每個client可能包含不同版本的方法調用。

clientset, err := kubernetes.NewForConfig(config)

2.3.1 NewForConfig

NewForConfig函數就是初始化clientset中的每個client。

k8s.io/client-go/kubernetes/clientset.go

// NewForConfig creates a new Clientset for the given config.func NewForConfig(c *rest.Config) (*Clientset, error) {    configShallowCopy := *c    ...    var cs Clientset    cs.appsV1beta1, err = appsv1beta1.NewForConfig(&configShallowCopy)    ...    cs.coreV1, err = corev1.NewForConfig(&configShallowCopy)    ...}

2.3.2 clientset的結構體

k8s.io/client-go/kubernetes/clientset.go

// Clientset contains the clients for groups. Each group has exactly one// version included in a Clientset.type Clientset struct {    *discovery.DiscoveryClient    admissionregistrationV1alpha1 *admissionregistrationv1alpha1.AdmissionregistrationV1alpha1Client    appsV1beta1                   *appsv1beta1.AppsV1beta1Client    appsV1beta2                   *appsv1beta2.AppsV1beta2Client    authenticationV1              *authenticationv1.AuthenticationV1Client    authenticationV1beta1         *authenticationv1beta1.AuthenticationV1beta1Client    authorizationV1               *authorizationv1.AuthorizationV1Client    authorizationV1beta1          *authorizationv1beta1.AuthorizationV1beta1Client    autoscalingV1                 *autoscalingv1.AutoscalingV1Client    autoscalingV2beta1            *autoscalingv2beta1.AutoscalingV2beta1Client    batchV1                       *batchv1.BatchV1Client    batchV1beta1                  *batchv1beta1.BatchV1beta1Client    batchV2alpha1                 *batchv2alpha1.BatchV2alpha1Client    certificatesV1beta1           *certificatesv1beta1.CertificatesV1beta1Client    coreV1                        *corev1.CoreV1Client    extensionsV1beta1             *extensionsv1beta1.ExtensionsV1beta1Client    networkingV1                  *networkingv1.NetworkingV1Client    policyV1beta1                 *policyv1beta1.PolicyV1beta1Client    rbacV1                        *rbacv1.RbacV1Client    rbacV1beta1                   *rbacv1beta1.RbacV1beta1Client    rbacV1alpha1                  *rbacv1alpha1.RbacV1alpha1Client    schedulingV1alpha1            *schedulingv1alpha1.SchedulingV1alpha1Client    settingsV1alpha1              *settingsv1alpha1.SettingsV1alpha1Client    storageV1beta1                *storagev1beta1.StorageV1beta1Client    storageV1                     *storagev1.StorageV1Client}

2.3.3 clientset.Interface

clientset實現了以下的Interface，因此可以通過調用以下方法獲得具體的client。例如：

pods, err := clientset.CoreV1().Pods("").List(metav1.ListOptions{})

clientset的方法集介面

k8s.io/client-go/kubernetes/clientset.go

type Interface interface {    Discovery() discovery.DiscoveryInterface    AdmissionregistrationV1alpha1() admissionregistrationv1alpha1.AdmissionregistrationV1alpha1Interface    // Deprecated: please explicitly pick a version if possible.    Admissionregistration() admissionregistrationv1alpha1.AdmissionregistrationV1alpha1Interface    AppsV1beta1() appsv1beta1.AppsV1beta1Interface    AppsV1beta2() appsv1beta2.AppsV1beta2Interface    // Deprecated: please explicitly pick a version if possible.    Apps() appsv1beta2.AppsV1beta2Interface    AuthenticationV1() authenticationv1.AuthenticationV1Interface    // Deprecated: please explicitly pick a version if possible.    Authentication() authenticationv1.AuthenticationV1Interface    AuthenticationV1beta1() authenticationv1beta1.AuthenticationV1beta1Interface    AuthorizationV1() authorizationv1.AuthorizationV1Interface    // Deprecated: please explicitly pick a version if possible.    Authorization() authorizationv1.AuthorizationV1Interface    AuthorizationV1beta1() authorizationv1beta1.AuthorizationV1beta1Interface    AutoscalingV1() autoscalingv1.AutoscalingV1Interface    // Deprecated: please explicitly pick a version if possible.    Autoscaling() autoscalingv1.AutoscalingV1Interface    AutoscalingV2beta1() autoscalingv2beta1.AutoscalingV2beta1Interface    BatchV1() batchv1.BatchV1Interface    // Deprecated: please explicitly pick a version if possible.    Batch() batchv1.BatchV1Interface    BatchV1beta1() batchv1beta1.BatchV1beta1Interface    BatchV2alpha1() batchv2alpha1.BatchV2alpha1Interface    CertificatesV1beta1() certificatesv1beta1.CertificatesV1beta1Interface    // Deprecated: please explicitly pick a version if possible.    Certificates() certificatesv1beta1.CertificatesV1beta1Interface    CoreV1() corev1.CoreV1Interface    // Deprecated: please explicitly pick a version if possible.    Core() corev1.CoreV1Interface    ExtensionsV1beta1() extensionsv1beta1.ExtensionsV1beta1Interface    // Deprecated: please explicitly pick a version if possible.    Extensions() extensionsv1beta1.ExtensionsV1beta1Interface    NetworkingV1() networkingv1.NetworkingV1Interface    // Deprecated: please explicitly pick a version if possible.    Networking() networkingv1.NetworkingV1Interface    PolicyV1beta1() policyv1beta1.PolicyV1beta1Interface    // Deprecated: please explicitly pick a version if possible.    Policy() policyv1beta1.PolicyV1beta1Interface    RbacV1() rbacv1.RbacV1Interface    // Deprecated: please explicitly pick a version if possible.    Rbac() rbacv1.RbacV1Interface    RbacV1beta1() rbacv1beta1.RbacV1beta1Interface    RbacV1alpha1() rbacv1alpha1.RbacV1alpha1Interface    SchedulingV1alpha1() schedulingv1alpha1.SchedulingV1alpha1Interface    // Deprecated: please explicitly pick a version if possible.    Scheduling() schedulingv1alpha1.SchedulingV1alpha1Interface    SettingsV1alpha1() settingsv1alpha1.SettingsV1alpha1Interface    // Deprecated: please explicitly pick a version if possible.    Settings() settingsv1alpha1.SettingsV1alpha1Interface    StorageV1beta1() storagev1beta1.StorageV1beta1Interface    StorageV1() storagev1.StorageV1Interface    // Deprecated: please explicitly pick a version if possible.    Storage() storagev1.StorageV1Interface}

2.4 CoreV1Client

我們以clientset中的CoreV1Client為例做分析。

通過傳入的配置資訊rest.Config初始化CoreV1Client對象。

k8s.io/client-go/kubernetes/clientset.go

cs.coreV1, err = corev1.NewForConfig(&configShallowCopy)

2.4.1 corev1.NewForConfig

k8s.io/client-go/kubernetes/typed/core/v1/core_client.go

// NewForConfig creates a new CoreV1Client for the given config.func NewForConfig(c *rest.Config) (*CoreV1Client, error) {    config := *c    if err := setConfigDefaults(&config); err != nil {        return nil, err    }    client, err := rest.RESTClientFor(&config)    if err != nil {        return nil, err    }    return &CoreV1Client{client}, nil}

corev1.NewForConfig方法本質是調用了rest.RESTClientFor(&config)方法建立RESTClient對象，即CoreV1Client的本質就是一個RESTClient對象。 2.4.2 CoreV1Client結構體

以下是CoreV1Client結構體的定義：

k8s.io/client-go/kubernetes/typed/core/v1/core_client.go

// CoreV1Client is used to interact with features provided by the  group.type CoreV1Client struct {    restClient rest.Interface}

CoreV1Client實現了CoreV1Interface的介面，即以下方法，從而對kubernetes的資來源物件進行增刪改查的操作。

k8s.io/client-go/kubernetes/typed/core/v1/core_client.go

//CoreV1Client的方法func (c *CoreV1Client) ComponentStatuses() ComponentStatusInterface {...}//ConfigMapsfunc (c *CoreV1Client) ConfigMaps(namespace string) ConfigMapInterface {...}//Endpointsfunc (c *CoreV1Client) Endpoints(namespace string) EndpointsInterface {...}func (c *CoreV1Client) Events(namespace string) EventInterface {...}func (c *CoreV1Client) LimitRanges(namespace string) LimitRangeInterface {...}//Namespacesfunc (c *CoreV1Client) Namespaces() NamespaceInterface {...}//Nodesfunc (c *CoreV1Client) Nodes() NodeInterface {...}func (c *CoreV1Client) PersistentVolumes() PersistentVolumeInterface {...}func (c *CoreV1Client) PersistentVolumeClaims(namespace string) PersistentVolumeClaimInterface {...}//Podsfunc (c *CoreV1Client) Pods(namespace string) PodInterface {...}func (c *CoreV1Client) PodTemplates(namespace string) PodTemplateInterface {...}//ReplicationControllersfunc (c *CoreV1Client) ReplicationControllers(namespace string) ReplicationControllerInterface {...}func (c *CoreV1Client) ResourceQuotas(namespace string) ResourceQuotaInterface {...}func (c *CoreV1Client) Secrets(namespace string) SecretInterface {...}//Servicesfunc (c *CoreV1Client) Services(namespace string) ServiceInterface {...}func (c *CoreV1Client) ServiceAccounts(namespace string) ServiceAccountInterface {...}

2.4.3 CoreV1Interface

k8s.io/client-go/kubernetes/typed/core/v1/core_client.go

type CoreV1Interface interface {    RESTClient() rest.Interface    ComponentStatusesGetter    ConfigMapsGetter    EndpointsGetter    EventsGetter    LimitRangesGetter    NamespacesGetter    NodesGetter    PersistentVolumesGetter    PersistentVolumeClaimsGetter    PodsGetter    PodTemplatesGetter    ReplicationControllersGetter    ResourceQuotasGetter    SecretsGetter    ServicesGetter    ServiceAccountsGetter}

CoreV1Interface中包含了各種kubernetes對象的調用介面，例如PodsGetter是對kubernetes中pod對象增刪改查操作的介面。ServicesGetter是對service對象的操作的介面。 2.4.4 PodsGetter

以下我們以PodsGetter介面為例分析CoreV1Client對pod對象的增刪改查介面調用。

樣本中的代碼如下：

pods, err := clientset.CoreV1().Pods("").List(metav1.ListOptions{})

CoreV1().Pods()

k8s.io/client-go/kubernetes/typed/core/v1/core_client.go

func (c *CoreV1Client) Pods(namespace string) PodInterface {    return newPods(c, namespace)}

newPods()

k8s.io/client-go/kubernetes/typed/core/v1/pod.go

// newPods returns a Podsfunc newPods(c *CoreV1Client, namespace string) *pods {    return &pods{        client: c.RESTClient(),        ns:     namespace,    }}

CoreV1().Pods()的方法實際上是調用了newPods()的方法，建立了一個pods對象，pods對象繼承了rest.Interface介面，即最終的實現本質是RESTClient的HTTP調用。

k8s.io/client-go/kubernetes/typed/core/v1/pod.go

// pods implements PodInterfacetype pods struct {    client rest.Interface    ns     string}

pods對象實現了PodInterface介面。PodInterface定義了pods對象的增刪改查等方法。

k8s.io/client-go/kubernetes/typed/core/v1/pod.go

// PodInterface has methods to work with Pod resources.type PodInterface interface {    Create(*v1.Pod) (*v1.Pod, error)    Update(*v1.Pod) (*v1.Pod, error)    UpdateStatus(*v1.Pod) (*v1.Pod, error)    Delete(name string, options *meta_v1.DeleteOptions) error    DeleteCollection(options *meta_v1.DeleteOptions, listOptions meta_v1.ListOptions) error    Get(name string, options meta_v1.GetOptions) (*v1.Pod, error)    List(opts meta_v1.ListOptions) (*v1.PodList, error)    Watch(opts meta_v1.ListOptions) (watch.Interface, error)    Patch(name string, pt types.PatchType, data []byte, subresources ...string) (result *v1.Pod, err error)    PodExpansion}

PodsGetter

PodsGetter繼承了PodInterface的介面。

k8s.io/client-go/kubernetes/typed/core/v1/pod.go

// PodsGetter has a method to return a PodInterface.// A group's client should implement this interface.type PodsGetter interface {    Pods(namespace string) PodInterface}

Pods().List()

pods.List()方法通過RESTClient的HTTP調用來實現對kubernetes的pod資源的擷取。

k8s.io/client-go/kubernetes/typed/core/v1/pod.go

// List takes label and field selectors, and returns the list of Pods that match those selectors.func (c *pods) List(opts meta_v1.ListOptions) (result *v1.PodList, err error) {    result = &v1.PodList{}    err = c.client.Get().        Namespace(c.ns).        Resource("pods").        VersionedParams(&opts, scheme.ParameterCodec).        Do().        Into(result)    return}

以上分析了clientset.CoreV1().Pods("").List(metav1.ListOptions{})對pod資源擷取的過程，最終是調用RESTClient的方法實現。 2.5 RESTClient

以下分析RESTClient的建立過程及作用。

RESTClient對象的建立同樣是依賴傳入的config資訊。

k8s.io/client-go/kubernetes/typed/core/v1/core_client.go

client, err := rest.RESTClientFor(&config)

2.5.1 rest.RESTClientFor

k8s.io/client-go/rest/config.go

// RESTClientFor returns a RESTClient that satisfies the requested attributes on a client Config// object. Note that a RESTClient may require fields that are optional when initializing a Client.// A RESTClient created by this method is generic - it expects to operate on an API that follows// the Kubernetes conventions, but may not be the Kubernetes API.func RESTClientFor(config *Config) (*RESTClient, error) {    ...    qps := config.QPS    ...    burst := config.Burst    ...    baseURL, versionedAPIPath, err := defaultServerUrlFor(config)    ...    transport, err := TransportFor(config)    ...    var httpClient *http.Client    if transport != http.DefaultTransport {        httpClient = &http.Client{Transport: transport}        if config.Timeout > 0 {            httpClient.Timeout = config.Timeout        }    }    return NewRESTClient(baseURL, versionedAPIPath, config.ContentConfig, qps, burst, config.RateLimiter, httpClient)}

RESTClientFor函數調用了NewRESTClient的初始化函數。 2.5.2 NewRESTClient

k8s.io/client-go/rest/client.go

// NewRESTClient creates a new RESTClient. This client performs generic REST functions// such as Get, Put, Post, and Delete on specified paths.  Codec controls encoding and// decoding of responses from the server.func NewRESTClient(baseURL *url.URL, versionedAPIPath string,