How to setup Kafka with SSL in Big Data environment | Kafka | Install | SSL

In this article, we will explain how to setup Kafka with SSL configurations. Here we provided Kafka commands for Linux and RHEL.



Kafka Installation with SSL Configurations

Kafka ssl setup (encryption between Kafka client and Kafka broker)

Step 1:First, we will create a CA Authority using below command

openssl req -new -newkey rsa:4096 -days 365 -x509 -subj "/CN=kafka-security-CA" -keyout ca-key -out ca-cert -nodes

Step 2: Setting up SSL for Kafka brokers in the cluster in Zookeeper:

I) Configure keystore for Kafka broker

keytool -genkey -keystore kafka1ks.jks -validity 365 -storepass sreekanth@123 -keypass sreekanth@123 -dname "CN=kafka1.training.com,OU=I/T Infrastructure,O=IBM INA Holdings Inc,L=Philadelphia,ST=Pennsylvania,C=US" -storetype pkcs12 --keyalg RSA -keysize 2048

b) Here to view the content of keystore:

keytool -list -keystore kafka1ks.jks -v

c) Get the signed version of keystore certificate (it is a two step process):

c.1) Do a certificate request from our keystore

keytool -keystore kafka1ks.jks -certreq -file kafka1ks.csr

c.2) Sign the requested certificate

openssl x509 -req -CA /home/kafka/ca/ca-cert -CAkey /home/kafka/ca/ca-key -in kafka1ks.csr -out kafka1ks.crt -days 365 -CAcreateserial

To list the signed certificate details:

keytool -printcert -v -file kafka1ks.crt

d) Now add the ca-cert (public certficate) and ca-signed(signed certificate) in kafka1ks.jks

keytool -keystore kafka1ks.jks -alias CARoot -import -file /home/kafka/ca/ca-cert -storepass sreekanth@123
(now ca-cert is added to the key store with alias CARoot)
keytool -keystore kafka1ks.jks -import -file kafka1ks.crt -storepass kafka1ks@123

(Now ca-signed cert is added to the key store)




II) create a trust store and import ca-cert (which is CA public certificate):

keytool -keystore kafka1ts.jks -alias CARoot -import -file /home/kafka/ca/ca-cert -storepass sreets@123

III) Using the certificates we have generated, we will setup kafka broker to use SSL on the port 9093
Alter server.properties file with below details:

listeners=PLAINTEXT://broker-host-name:non-ssl-port,SSL://broker-host-name:ssl-port
advertised.listeners=PLAINTEXT://broker-host-name:non-ssl-port,SSL://broker-host-name:ssl-port
ssl.keystore.location=/home/apps/kafka/private/ssl/server.keystore.jks
ssl.keystore.password=<keystore-password>
ssl.key.password=<keystore-password>
ssl.truststore.location=/home/apps/kafka/private/ssl/server.truststore.jks
ssl.truststore.password=<keystore-password>

IV) Once kafka is rebooted, we will test the SSL certificates once again to make sure they are correctly applied
To test the SSL connectivity from your kafka client to brokers:

openssl s_client -connect <broker-host-name>:<broker-ssl-port>

Step 3:




Kafka Client side setup:
————————-
I) Get the ca-cert (CA public certificate) from kafka broker server to client server (application server)

II) Create a trust store for kafka client server and import ca-cert file in trust store

keytool -keystore maheshts.jks -alias caroot -import -file ca-cert

III) Update the client.properties file with below values

security.protocol=SSL
ssl.truststore.location=<location-of-client-truststore-file-name>.jks
ssl.truststore.password=<truststore-password>

After completed above steps then will send messages from kafka client in secure mode.