Kafka with TLS
This page explains how to create TLS certificates to secure data using Kafka.
Refer to the Example Overview for details about this example.
Kafka supports encrypting data in transit using transport layer security (TLS) encryption. To configure TLS encryption, the Kafka broker must be configured with TLS settings enabled. See the Confluent guide on authentication for a guide on configuring the broker with a key and certificate to enable TLS encryption.
Note
TLS is the newer version of what was previously referred to as secure socket layer (SSL) encryption. The guide above refers to SSL authentication which can be used interchangeably with TLS.
Creating self-signed certificates
This section walks through creating self-signed certificates to use with the broker and consumers. It is recommended that in production an official certificate authority is used to issue these certificates.
Note
When creating certificates, most require a password. It is recommended that a strong password is used and the same password is used for the key and certificate. When configuring the common name (CN) for the certificate, use the external hostname of the target Kafka broker instance when configuring this value.
Typically brokers are configured with a keystore.jks and truststore.jks file. The Java
Keystore (JKS) format is a proprietary Java format that is not compatible with the Kafka
reader. These certificates can be converted into Rivest-Shamir-Adleman (RSA) encoded
Privacy Enhanced Mail (PEM) files. This section walks through converting a keystore and
truststore into a certificate authority (CA) root file, a key and a certificate encoded
as .pem files.
1. Generate the keystore
Bash
# keytool is bundled with the JDK and is used to generate the keystore for this example.
#
# -keystore <filename> - output file
# -alias <hostname> - alias name from the entry process
# -validity <days> - valid duration of this keystore in days
keytool -genkey -keyalg RSA -keystore keystore.jks -alias localhost -validity 365
2. Generate the certificate authority
The CA file is simply a public-private key pair certificate that is used to sign other certificates. The CA file needs to be installed in the server key and client certificate to validate the authenticity.
Bash
openssl req -new -x509 -keyout ca-key.pem -out ca-cert.pem -days 365
3. Create the truststore from the certificate authority
Generate the truststore for the server using the certificate authority.
Bash
keytool -import -keystore truststore.jks -alias CARoot -file ca-cert.pem -trustcacerts
4. Sign the certificates with the certificate authority
Export the certificate from the keystore.
Bash
keytool -certreq -keystore keystore.jks -alias localhost -file unsigned.pem -trustcacerts
Sign the certificate with the CA file.
Bash
openssl x509 -req -CA ca-cert.pem -CAkey ca-key.pem -in unsigned.pem -out signed.pem -days 365 -CAcreateserial
Import the signed certificate into the server keystore for each alias.
Bash
keytool -import -keystore keystore.jks -alias CARoot -file ca-cert.pem -trustcacerts
Bash
keytool -import -keystore keystore.jks -alias localhost -file signed.pem -trustcacerts
5. Export the client PEM files
Export the client certificate from the keystore.
Bash
keytool -exportcert -alias localhost -keystore keystore.jks -rfc -file cert.pem
Export the key to a Public Key Cryptography Standards 12 (PKCS12) format. This is used as an intermediary format which is converted to a PEM file.
Bash
keytool -importkeystore \
-srckeystore keystore.jks \
-destkeystore keystore.p12 \
-deststoretype PKCS12 \
-srcalias localhost
Extract the key from the PKCS12 file into a PEM file.
Bash
openssl pkcs12 -in keystore.p12 -nocerts -nodes -out key.pem
6. Cleanup intermediate files
A number of the files generated were intermediate files and can now be removed
Bash
rm ca-key.pem unsigned.pem signed.pem keystore.p12
Summary
At this point we have created the following files.
|
file |
description |
|---|---|
|
keystore.jks |
Kafka broker keystore |
|
truststore.jks |
Kafka broker truststore |
|
ca-cert.pem |
CA certificate |
|
cert.pem |
Client certificate |
|
key.pem |
Client key |
These files are used in later steps to configure the Kafka client to read from the Kafka broker.
TLS Kafka reader in the Stream Processor
TLS configuration can be set on a Kafka reader or writer by explicitly setting the configuration values in a program file or with environment variables. The following table outlines the available variables that can be used for configuration. For more information on these configuration properties, refer to the librdkafka documentation
|
environment variable |
property |
description |
example |
|---|---|---|---|
|
|
|
Security communication protocol |
|
|
|
|
CA PEM file location |
|
|
|
|
Client certificate location |
|
|
|
|
Client key location |
|
|
|
|
Client key password |
|
|
|
|
Set to |
|
Example program called spec.q:
q
.qsp.run
.qsp.read.fromKafka[.qsp.use (!) . flip (
(`brokers ; "kafka:9092");
(`topic ; "trades");
(`options; (!) . flip (
(`security.protocol ; "SSL");
(`ssl.ca.location ; "/certs/ca-cert.pem");
(`ssl.certificate.location ; "/certs/cert.pem");
(`ssl.key.location ; "/certs/key.pem");
(`ssl.key.password ; "iamsecure"))))]
.qsp.decode.json[]
.qsp.map[{ flip `time`sym`bid`ask!"PSff"$'flip x }]
.qsp.window.timer[00:00:10]
.qsp.write.toProcess[.qsp.use `handle`mode`target!(`:localhost:4000; `function; `publish)]