golang用戶端sarama通過SSL串連Kafka配置

這是一個建立於 的文章，其中的資訊可能已經有所發展或是發生改變。

• 前言

非對稱金鑰、認證、簽名、keystone、truststore等相關概念請移步度娘查詢，在此只記錄下相關步驟

• 認證產生

#!/bin/shkeytool -keystore kafka.server.keystore.jks -alias localhost -validity 3650 -keyalg RSA -genkeyopenssl req -new -x509 -keyout ca-key -out ca-cert -days 3650keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-certkeytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-certkeytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-fileopenssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 3650 -CAcreateserial -passin pass:123456keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-certkeytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signedkeytool -importkeystore -srckeystore kafka.server.truststore.jks -destkeystore server.p12 -deststoretype PKCS12openssl pkcs12 -in server.p12 -nokeys -out server.cer.pemkeytool -importkeystore -srckeystore kafka.server.keystore.jks -destkeystore client.p12 -deststoretype PKCS12openssl pkcs12 -in client.p12 -nokeys -out client.cer.pemopenssl pkcs12 -in client.p12 -nodes -nocerts -out client.key.pem

• 服務端配置

通過上面執行的指令碼，Kafka的broker使用kafka.server.truststore.jks和kafka.server.keystore.jks，修改設定檔server.properties

 listeners=PLAINTEXT://x.x.x.x:9092,SSL://x.x.x.x:9093 ssl.keystore.location=kafka.server.keystore.jks ssl.keystore.password=123456 ssl.key.password=123456 ssl.truststore.location=kafka.server.truststore.jks ssl.truststore.password=123456 ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 ssl.keystore.type=JKS ssl.truststore.type=JKS

• 用戶端配置

用戶端會用到上面產生的server.cer.pem，client.cer.pem（重點：需要修改），client.key.pemclient.cer.pem，client.key.pem可能出現"private key does not match public key"錯誤，那麼需要手動處理client.cer.pem檔案

Bag Attributes    friendlyName: caroot    2.16.840.1.xxx.xxx.1.1: <Unsupported tag 6>subject=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xxissuer=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx-----BEGIN CERTIFICATE-----MIIDgzCCAmugAwIBAgIJAL95jWSrh9jfMA0GCSqGSIb3DQEBCwUAMFgxCzAJBgNVBAYTAmNuMQswCQYDVQQIDAJiajELMAkGA1UEBwwCYmoxCzAJBgNVBAoMAnp3MQswCQYDVQQLDAJ6dzEVMBMGA1UEAwwMMTkyLjE2OC4yLjMxMB4XDTE4MDMxOTEyMTAxOVoXDTI4MDMxNjEyMTAxOVowW9....省略n個字元-----END CERTIFICATE-----Bag Attributes    friendlyName: localhost    localKeyID: 54 69 6D 65 20 31 35 32 31 34 36 31 34 34 36 xx xx xxsubject=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xxissuer=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx-----BEGIN CERTIFICATE-----MIIDLDCCAhQCCQDqwOxGdLTDLjANBgkqhkiG9w0BAQUFADBYMQswCQYDVQQGEwJjbjELMAkGA1UECAwCYmoxCzAJBgNVBAcMAmJqMQswCQYDVQQKDAJ6dzELMAkGA1UECwwCencxFTATBgNVBAMMDDE5Mi4xNjguMi4zMTAeFw0xODAzMTkxMjEwMzBaFw0yODAzMTYxMjEwMzBaMFgxCzAJBgNVBAYTAmNuMQswCQYDVQQIEwJiajELMAkGA1UEBxMCYmoxCzAJBgNVBAoTAnp3MQswCQYDVQQLEwJ6dzEVMBMGA1UEAxMMMTkyLjE2OC4yLjMxMIIBIjANBgkqhkiG9w0BAQ....省略n個字元-----END CERTIFICATE-----Bag Attributes    friendlyName: CN=192.168.xx.xx,OU=xx,O=xx,L=bj,ST=bj,C=cnsubject=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xxissuer=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx-----BEGIN CERTIFICATE-----MIIDgzCCAmugAwIBAgIJAL95jWSrh9jfMA0GCSqGSIb3DQEBCwUAMFgxCzAJBgNVBAYTAmNuMQswCQYDVQQIDAJiajELMAkGA1UEBwwCYmoxCzAJBgNVBAoMAnp3MQswCQYDVQQLDAJ6dzEVMBMGA1UEAwwMMTkyLjE2OC4yLjMxMB4XDTE4MDMxOTEyMTAxOVoXDTI4MDMxNjEyMTAxOVowWDELMAkGA1UEBhMCY24xCzAJBgNVBAgMAmJqMQswCQYDVQQHDAJiajELMAkGA1UECg....省略n個字元-----END CERTIFICATE-----

此處查看到有三段-----BEGIN CERTIFICATE----- 和 -----END CERTIFICATE-----，開啟client.key.pem看到只有一段friendlyName: localhost，那麼找到client.cer.pem（為中間一段），刪除其餘部分，剩餘如下：

-----BEGIN CERTIFICATE-----MIIDLDCCAhQCCQDqwOxGdLTDLjANBgkqhkiG9w0BAQUFADBYMQswCQYDVQQGEwJjbjELMAkGA1UECAwCYmoxCzAJBgNVBAcMAmJqMQswCQYDVQQKDAJ6dzELMAkGA1UECwwCencxFTATBgNVBAMMDDE5Mi4xNjguMi4zMTAeFw0xODAzMTkxMjEwMzBaFw0yODAzMTYxMjEwMzBaMFgxCzAJBgNVBAYTAmNuMQswCQYDVQQIEwJiajELMAkGA1UEBxMCYmoxCzAJBgNVBAoTAnp3MQswCQYDVQQLEwJ6dzEVMBMGA1UEAxMMMTkyLjE2OC4yLjMxMIIBIjANBgkqhkiG9w0BAQ....省略n個字元-----END CERTIFICATE-----

• 測試程式如下：

package main// Run with:// go build examples/base-client/*.go// ./base-clientimport ("crypto/tls""crypto/x509""io/ioutil""log""os""os/signal""sync""github.com/Shopify/sarama")func main() {tlsConfig, err := NewTLSConfig("bundle/client.cer.pem","bundle/client.key.pem","bundle/server.cer.pem")if err != nil {log.Fatal(err)}// This can be used on test server if domain does not match cert:tlsConfig.InsecureSkipVerify = trueconsumerConfig := sarama.NewConfig()consumerConfig.Net.TLS.Enable = trueconsumerConfig.Net.TLS.Config = tlsConfigclient, err := sarama.NewClient([]string{"192.168.2.31:9093"}, consumerConfig)if err != nil {log.Fatalf("unable to create kafka client: %q", err)}consumer, err := sarama.NewConsumerFromClient(client)if err != nil {log.Fatal(err)}defer consumer.Close()consumerLoop(consumer, "Test")}// NewTLSConfig generates a TLS configuration used to authenticate on server with// certificates.// Parameters are the three pem files path we need to authenticate: client cert, client key and CA cert.func NewTLSConfig(clientCertFile, clientKeyFile, caCertFile string) (*tls.Config, error) {tlsConfig := tls.Config{}// Load client certcert, err := tls.LoadX509KeyPair(clientCertFile, clientKeyFile)if err != nil {return &tlsConfig, err}tlsConfig.Certificates = []tls.Certificate{cert}// Load CA certcaCert, err := ioutil.ReadFile(caCertFile)if err != nil {return &tlsConfig, err}caCertPool := x509.NewCertPool()caCertPool.AppendCertsFromPEM(caCert)tlsConfig.RootCAs = caCertPooltlsConfig.BuildNameToCertificate()return &tlsConfig, err}func consumerLoop(consumer sarama.Consumer, topic string) {partitions, err := consumer.Partitions(topic)if err != nil {log.Println("unable to fetch partition IDs for the topic", topic, err)return}// Trap SIGINT to trigger a shutdown.signals := make(chan os.Signal, 1)signal.Notify(signals, os.Interrupt)var wg sync.WaitGroupfor partition := range partitions {wg.Add(1)go func() {consumePartition(consumer, int32(partition), signals)wg.Done()}()}wg.Wait()}func consumePartition(consumer sarama.Consumer, partition int32, signals chan os.Signal) {log.Println("Receving on partition", partition)partitionConsumer, err := consumer.ConsumePartition("test", partition, sarama.OffsetNewest)if err != nil {log.Println(err)return}defer func() {if err := partitionConsumer.Close(); err != nil {log.Println(err)}}()consumed := 0ConsumerLoop:for {select {case msg := <-partitionConsumer.Messages():log.Printf("Consumed message offset %d\nData: %s\n", msg.Offset, msg.Value)consumed++case <-signals:break ConsumerLoop}}log.Printf("Consumed: %d\n", consumed)}

按照網上的教程出現各種錯誤，主要在於client.cer.pem需要手動修改

參考地址：

https://docs.confluent.io/current/tutorials/security_tutorial.html#creating-ssl-keys-and-certificates

https://medium.com/processone/using-tls-authentication-for-your-go-kafka-client-3c5841f2a625

https://github.com/FluuxIO/kafka/blob/master/examples/base-client/base-client.go