Golang client Sarama via SSL connection Kafka configuration

This is a creation in Article, where the information may have evolved or changed.

Objective

Asymmetric key, certificate, signature, Keystone, Truststore and other related concepts please visit the mother of the query, only record the relevant steps

Certificate generation

#!/bin/sh
keytool -keystore kafka.server.keystore.jks -alias localhost -validity 3650 -keyalg RSA -genkey
openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert

keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 3650 -CAcreateserial -passin pass:123456
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signed

keytool -importkeystore -srckeystore kafka.server.truststore.jks -destkeystore server.p12 -deststoretype PKCS12
openssl pkcs12 -in server.p12 -nokeys -out server.cer.pem
keytool -importkeystore -srckeystore kafka.server.keystore.jks -destkeystore client.p12 -deststoretype PKCS12
openssl pkcs12 -in client.p12 -nokeys -out client.cer.pem
openssl pkcs12 -in client.p12 -nodes -nocerts -out client.key.pem

Server-side configuration
With the script executed above, Kafka's broker uses KAFKA.SERVER.TRUSTSTORE.JKS and Kafka.server.keystore.jks to modify the configuration file Server.properties

listeners=PLAINTEXT://x.x.x.x:9092,SSL://x.x.x.x:9093
 ssl.keystore.location=kafka.server.keystore.jks
 ssl.keystore.password=123456
 ssl.key.password=123456
 ssl.truststore.location=kafka.server.truststore.jks
 ssl.truststore.password=123456
 ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
 ssl.keystore.type=JKS
 ssl.truststore.type=JKS

Client Configuration

The client will use the SERVER.CER.PEM,CLIENT.CER.PEM generated above (emphasis: needs to be modified), CLIENT.KEY.PEMCLIENT.CER.PEM,CLIENT.KEY.PEM may appear "private key Does not match public key "error, then manual processing of CLIENT.CER.PEM file is required

Bag Attributes
    friendlyName: caroot
    2.16.840.1.xxx.xxx.1.1: <Unsupported tag 6>
Subject=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
Issuer=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
-----BEGIN CERTIFICATE-----
MIIDgzCCAmugAwIBAgIJAL95jWSrh9jfMA0GCSqGSIb3DQEBCwUAMFgxCzAJBgNV
BAYTAmNuMQswCQYDVQQIDAJiajELMAkGA1UEBwwCYmoxCzAJBgNVBAoMAnp3MQsw
CQYDVQQLDAJ6dzEVMBMGA1UEAwwMMTkyLjE2OC4yLjMxMB4XDTE4MDMxOTEyMTAx
OVoXDTI4MDMxNjEyMTAxOVowW9....omitting n characters
-----END CERTIFICATE-----
Bag Attributes
    friendlyName: localhost
    localKeyID: 54 69 6D 65 20 31 35 32 31 34 36 31 34 34 36 xx xx xx
Subject=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
Issuer=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
-----BEGIN CERTIFICATE-----
MIIDLDCCAhQCCQDqwOxGdLTDLjANBgkqhkiG9w0BAQUFADBYMQswCQYDVQQGEwJj
bjELMAkGA1UECAwCYmoxCzAJBgNVBAcMAmJqMQswCQYDVQQKDAJ6dzELMAkGA1UE
CwwCencxFTATBgNVBAMMDDE5Mi4xNjguMi4zMTAeFw0xODAzMTkxMjEwMzBaFw0y
ODAzMTYxMjEwMzBaMFgxCzAJBgNVBAYTAmNuMQswCQYDVQQIEwJiajELMAkGA1UE
BxMCYmoxCzAJBgNVBAoTAnp3MQswCQYDVQQLEwJ6dzEVMBMGA1UEAxMMMTkyLjE2
OC4yLjMxMIIBIjANBgkqhkiG9w0BAQ....omits n characters
-----END CERTIFICATE-----
Bag Attributes
    friendlyName: CN=192.168.xx.xx, OU=xx, O=xx, L=bj, ST=bj, C=cn
Subject=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
Issuer=/C=cn/ST=bj/L=bj/O=xx/OU=xx/CN=192.168.xx.xx
-----BEGIN CERTIFICATE-----
MIIDgzCCAmugAwIBAgIJAL95jWSrh9jfMA0GCSqGSIb3DQEBCwUAMFgxCzAJBgNV
BAYTAmNuMQswCQYDVQQIDAJiajELMAkGA1UEBwwCYmoxCzAJBgNVBAoMAnp3MQsw
CQYDVQQLDAJ6dzEVMBMGA1UEAwwMMTkyLjE2OC4yLjMxMB4XDTE4MDMxOTEyMTAx
OVoXDTI4MDMxNjEyMTAxOVowWDELMAkGA1UEBhMCY24xCzAJBgNVBAgMAmJqMQsw
CQYDVQQHDAJiajELMAkGA1UECg....omits n characters
-----END CERTIFICATE-----

Here you see there are three paragraphs-----BEGIN CERTIFICATE-----and-----END CERTIFICATE-----, open CLIENT.KEY.PEM see only a friendlyname:localhost, Then find Client.cer.pem (for the middle section), delete the remainder, and the remainder is as follows:

-----BEGIN CERTIFICATE-----
MIIDLDCCAhQCCQDqwOxGdLTDLjANBgkqhkiG9w0BAQUFADBYMQswCQYDVQQGEwJj
bjELMAkGA1UECAwCYmoxCzAJBgNVBAcMAmJqMQswCQYDVQQKDAJ6dzELMAkGA1UE
CwwCencxFTATBgNVBAMMDDE5Mi4xNjguMi4zMTAeFw0xODAzMTkxMjEwMzBaFw0y
ODAzMTYxMjEwMzBaMFgxCzAJBgNVBAYTAmNuMQswCQYDVQQIEwJiajELMAkGA1UE
BxMCYmoxCzAJBgNVBAoTAnp3MQswCQYDVQQLEwJ6dzEVMBMGA1UEAxMMMTkyLjE2
OC4yLjMxMIIBIjANBgkqhkiG9w0BAQ....omits n characters
-----END CERTIFICATE-----

• The test procedure is as follows:

package main

// Run with:
// go build examples/base-client/*.go
// ./base-client

import (
	"crypto/tls"
	"crypto/x509"
	"io/ioutil"
	"log"
	"os"
	"os/signal"
	"sync"
	"github.com/Shopify/sarama"
)

func main() {
	tlsConfig, err := NewTLSConfig("bundle/client.cer.pem",
		"bundle/client.key.pem",
		"bundle/server.cer.pem")
	if err != nil {
		log.Fatal(err)
	}
	// This can be used on test server if domain does not match cert:
	tlsConfig.InsecureSkipVerify = true

	consumerConfig := sarama.NewConfig()
	consumerConfig.Net.TLS.Enable = true
	consumerConfig.Net.TLS.Config = tlsConfig

	client, err := sarama.NewClient([]string{"192.168.2.31:9093"}, consumerConfig)
	if err != nil {
		log.Fatalf("unable to create kafka client: %q", err)
	}

	consumer, err := sarama.NewConsumerFromClient(client)
	if err != nil {
		log.Fatal(err)
	}
	defer consumer.Close()

	consumerLoop(consumer, "Test")
}

// NewTLSConfig generates a TLS configuration used to authenticate on server with
// certificates.
// Parameters are the three pem files path we need to authenticate: client cert, client key and CA cert.
func NewTLSConfig(clientCertFile, clientKeyFile, caCertFile string) (*tls.Config, error) {
	tlsConfig := tls.Config{}

	// Load client cert
	cert, err := tls.LoadX509KeyPair(clientCertFile, clientKeyFile)
	if err != nil {
		return &tlsConfig, err
	}
	tlsConfig.Certificates = []tls.Certificate{cert}

	// Load CA cert
	caCert, err := ioutil.ReadFile(caCertFile)
	if err != nil {
		return &tlsConfig, err
	}
	caCertPool := x509.NewCertPool()
	caCertPool.AppendCertsFromPEM(caCert)
	tlsConfig.RootCAs = caCertPool

	tlsConfig.BuildNameToCertificate()
	return &tlsConfig, err
}

func consumerLoop(consumer sarama.Consumer, topic string) {
	partitions, err := consumer.Partitions(topic)
	if err != nil {
		log.Println("unable to fetch partition IDs for the topic", topic, err)
		return
	}

	// Trap SIGINT to trigger a shutdown.
	signals := make(chan os.Signal, 1)
	signal.Notify(signals, os.Interrupt)

	var wg sync.WaitGroup
	for partition := range partitions {
		wg.Add(1)
		go func() {
			consumePartition(consumer, int32(partition), signals)
			wg.Done()
		}()
	}
	wg.Wait()
}

func consumePartition(consumer sarama.Consumer, partition int32, signals chan os.Signal) {
	log.Println("Receving on partition", partition)
	partitionConsumer, err := consumer.ConsumePartition("test", partition, sarama.OffsetNewest)
	if err != nil {
		log.Println(err)
		return
	}
	defer func() {
		if err := partitionConsumer.Close(); err != nil {
			log.Println(err)
		}
	}()

	consumed := 0
ConsumerLoop:
	for {
		select {
		case msg := <-partitionConsumer.Messages():
			log.Printf("Consumed message offset %d\nData: %s\n", msg.Offset, msg.Value)
			consumed++
		case <-signals:
			break ConsumerLoop
		}
	}
	log.Printf("Consumed: %d\n", consumed)
}
 

There are various errors in the online tutorials, mainly because CLIENT.CER.PEM needs to be manually modified

Reference Address:


Https://docs.confluent.io/current/tutorials/security_tutorial.html#creating-ssl-keys-and-certificates

https://medium.com/processone/using-tls-authentication-for-your-go-kafka-client-3c5841f2a625

Https://github.com/FluuxIO/kafka/blob/master/examples/base-client/base-client.go

Alibaba Cloud Hot Products
Elastic Compute Service (ECS)	Dedicated Host (DDH)	ApsaraDB RDS for MySQL (RDS)	ApsaraDB for PolarDB(PolarDB)	AnalyticDB for PostgreSQL (ADB for PG)
AnalyticDB for MySQL(ADB for MySQL)	Data Transmission Service (DTS)	Server Load Balancer (SLB)	Global Accelerator (GA)	Cloud Enterprise Network (CEN)
Object Storage Service (OSS)	Content Delivery Network (CDN)	Short Message Service (SMS)	Container Service for Kubernetes (ACK)	Data Lake Analytics (DLA)
ApsaraDB for Redis (Redis)	ApsaraDB for MongoDB (MongoDB)	NAT Gateway	VPN Gateway	Cloud Firewall
Anti-DDoS	Web Application Firewall (WAF)	Log Service	DataWorks	MaxCompute
Elastic MapReduce (EMR)	Elasticsearch