Setting Filebeat 5 with Kafka 0.10 over TLS

Tsury · September 1, 2016, 1:05pm

I have a perfectly working Filebeat 5.0.0-alpha5 and Kafka 0.10 setup. They reside in different physical servers, each as a Docker container.

I am trying to set them up to use SSL(TLS) instead of PLAINTEXT.

I am just testing it for now so I don't have any previous certificates/CAs and am creating everything now for my development environment.

I started by following Apache's documentation about how to set up TLS for Kafka.

Basically it amounted to this bash script:

#!/bin/bash
PASSWORD=test1234
VALIDITY=365
keytool -keystore kafka.server.keystore.jks -alias localhost -validity $VALIDITY -genkey
openssl req -new -x509 -keyout ca-key -out ca-cert -days $VALIDITY
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days $VALIDITY -CAcreateserial -passin pass:$PASSWORD
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signed
keytool -keystore kafka.client.keystore.jks -alias localhost -validity $VALIDITY -genkey
keytool -keystore kafka.client.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days $VALIDITY -CAcreateserial -passin pass:$PASSWORD
keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore kafka.client.keystore.jks -alias localhost -import -file cert-signed

Which produced the following files:

ca-cert
ca-cert.srl
ca-key
cert-file
cert-signed
kafka.client.keystore.jks
kafka.client.truststore.jks
kafka.server.keystore.jks
kafka.server.truststore.jks

After setting up Kafka's config files and restarting it, everything seems to be working fine.

Now begins the part where I am unsure of what I'm doing.

I know that Filebeats' TLS settings requires these 3 entries:

  tls.certificate_authorities: ["file"]
  tls.certificate: "file"
  tls.certificate_key: "file"

I tried to identify my files and match them, and guessed that ca-cert is for tls.certificate_authorities and cert_signed is for tls.certificate. Is this correct?

This is how they look:

ca_cert:

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

cert_signed:

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Now I tried to extract the private key from kafka.client.keystore.jks by saving it as a p12 file:

keytool -importkeystore -srckeystore kafka.client.keystore.jks -destkeystore kafka.client.keystore.p12 -deststoretype PKCS12 -srcalias localhost -deststorepass test1234 -destkeypass test1234
openssl pkcs12 -in kafka.client.keystore.p12  -nodes -nocerts -out key.pem

key.pem:

Bag Attributes
    friendlyName: localhost
    localKeyID: SOME HEX
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----

Now that I have all the files, I copied them to the Filebeat server and tried to start it with these settings under the Kafka output section:

  tls.certificate_authorities: ["/ssl/ca-cert"]
  tls.certificate: "/ssl/cert-signed"
  tls.certificate_key: "/ssl/key.pem"

But then Filebeat exits instantly with this error:

filebeat_1     | 2016/08/31 16:02:41.276236 beat.go:263: INFO Home path: [/] Config path: [/] Data path: [//data] Logs path: [//logs]
filebeat_1     | 2016/08/31 16:02:41.284449 beat.go:174: INFO Setup Beat: filebeat; Version: 5.0.0-alpha5
filebeat_1     | 2016/08/31 16:02:41.284464 processor.go:42: DBG  Processors:
filebeat_1     | 2016/08/31 16:02:41.284470 beat.go:180: DBG  Initializing output plugins
filebeat_1     | 2016/08/31 16:02:41.284494 kafka.go:67: DBG  initialize kafka output
filebeat_1     | 2016/08/31 16:02:41.287109 tls.go:98: CRIT Failed loading client certificate%!(EXTRA *errors.errorString=crypto/tls: failed to parse private key)
filebeat_1     | 2016/08/31 16:02:41.287123 outputs.go:81: ERR failed to initialize kafka plugin as output: crypto/tls: failed to parse private key
filebeat_1     | 2016/08/31 16:02:41.287129 beat.go:284: CRIT Exiting: error initializing publisher: crypto/tls: failed to parse private key
filebeat_1     | Exiting: error initializing publisher: crypto/tls: failed to parse private key

I also tried to crop the 'header' of the pem file to make it look like this:

-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----

Still no cigar.

What am I missing here?
I would appreciate any help at this point, thanks!

steffens · September 2, 2016, 2:07pm

Setting up certificates chains with mix of openssl and keytool can be super painful. I'd say let's do it step by step and do TLS with server certificate only first. Once this works, we can add client authentication.

For TLS to work, the server normally requires a certificate + private key. The client only requires the trustchain/truststore/ca-pool used for validation. The truststore is a set of public certificates we do trust. If server certificate is self-signed, the truststore will contain the server certificate. Otherwise it will contain the certificates used to sign the server certificate. In your case ca-cert I think.

Also note, the server certificates hostname used in subject must match the hostname client uses for connecting to. This is a minor pitfall with kafka due to: kafka advertising it's host (configurable) either by hostname or ip. It might even advertise the wrong hostname (a hostname not matching your certificate). If everything is running in same docker instance, the hostname available to client docker container should be the container-name used by kafka.

Having server certificate+key + trustchain we can setup kafka to enable TLS:

server.properties

listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
ssl.keystore.location=<path-to>/kafka.server.keystore.jks
ssl.keystore.password=password
ssl.key.password=password

Note: The PLAINTEXT socket is still open. It's nice for testing later on, as the kafka CLI test consumer can connect via this port tailing a topic we will have filebeat to push to via TLS. Remove PLAINTEXT later if not required anymore.

Starting kafka with this config we can use openssl s_client verifying everything is setup correctly so far:

$ openssl s_client --connect <domain>:9093 -CAfile  <path-to-trustchain>
...
^D
$ echo $?
0

having only one certificate, the trustchain should equal ca-cert.

With kafka being available via docker I'd recommend doing the openssl from withing docker container (e.g. ubuntu image linked to kafka container).

If this works ok, let's configure filebeat to enable tls (without client authentication):

output.kafka:
  hosts: ['kafka:9093']
  tls:
    certificate_authorities:
      - '<path-to-trustchain>'

.

Next let's add client authentication (Note, only with upcoming beta1, filebeat will support encrypted private key). Client authentication is optional, TLS with server certificate is already active. To me it looks like your usage of pkcs12 does not create a valid key.pem file (you've got a buggy version of openssl?). Creating a pkcs12 store from key generated via openssl and extracting the private key again works for me (diff of extracted vs. original gives me no differences besides all content before -----BEGIN PRIVATE KEY-----).
Note: I can't really tell from you'r example which certificate/key is for client and which one is for server. You should use different certificates for each.
Note: if you're on mac, get openssl from homebrew and use /usr/local/openssl/bin/openssl instead of system one (I sometimes have had problems with openssl as provided by system).

Given we've got a valid client certificate + key file + trustchain for client we can configure kafka accordingly:

server.properties

listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
ssl.keystore.location=<path-to>/kafka.server.keystore.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.truststore.location=<path-to>/kafka.server.truststore.jks
ssl.truststore.password=password
ssl.client.auth=required

We did add the truststore + enabled client authentication in kafka. Next let's verify with openssl:

$ openssl s_client -connect localhost:5066 -showcerts -CAfile <path-to>/trustchain.cert.pem -cert <path-to>/client.cert.pem -key ssl/private/client.key.pem
...
^D
$ echo $?
0

With openssl being able to connect, we can add client authentication to filebeat:

output.kafka:
  hosts: ['kafka:9093']
  tls:
    certificate: '<path-to>/client.cert.pem'
    certificate_key: '<path-to>/client.key.pem'
    certificate_authorities:
      - '<path-to-trustchain>'

Personally I find using keytool for setting all this up a little tricky. I prefer building all certificates using openssl and finally using a mix of openssl pkcs12 and keytool to create keystore and truststore for use with java services.

This gist contains makefile + openssl configuration files to create CA-Cert, intermediate CA-cert + client and server certificates for use with any service (update OPENSSL variable in script to point to openssl instance you want to use).

I hope this helps.

anefassa · September 2, 2016, 6:46pm

When I tried the bash script you provided, which I see just follows the directions on Kafka docs, I got the same error in filebeat.

Exiting: error initializing publisher: crypto/tls: failed to parse private key

Looking into it I noticed that the script generated DSA keys for the server and client rather than RSA keys. The generated key worked okay from openssl but failed from filebeat. From my tests it looks like filebeat does not support DSA keys and will fail if one is used in either the client or the server.

Additionally, as Steffen pointed out, when generating the server certificate I needed to provide the exact Kafka hostname that was going to be used by the client. Possibly adding
insecure
to the tls options in the Kafka output config might make filebeat accept a mismatch on the hostname and cert CN. Refer to tls config docs for information on the insecure option but one should not use insecure outside of testing due to potential man-in-the-middle attacks.

I created a new bash script to create a self-signed RootCA, create the required server and client certificates, keystores, truststores and these worked for me. Will provide the script on a separate reply as it will be too large to add here.

anefassa · September 2, 2016, 6:48pm

I created a new bash script to create a self-signed RootCA, create the required server and client certificates, keystores, truststores and these worked for me.

#!/bin/bash
PASSWORD=password
VALIDITY=365
SERVER_HOSTNAME=YourServerHostname


# Step 1 - Generate CA cert
echo ""
echo "##############"
echo "Will Create key pair for Root CA, to be used for signing other certs"
echo "##############"
read -n1 -rsp $'Press any key to continue or Ctrl+C to exit...\n'
openssl req -new -x509 -keyout rootca.key.pem -out rootca.cert.pem -days $VALIDITY

# Generate server keystore with RSA private key
echo ""
echo "##############"
echo "Will create jks with server private key in RSA format"
echo "***** When asked for first and last name"
echo "***** Make sure to provide the hostname client uses for connecting"
echo "##############"
read -n1 -rsp $'Press any key to continue or Ctrl+C to exit...\n'
keytool -genkey -alias $SERVER_HOSTNAME -keyalg RSA -keystore $SERVER_HOSTNAME-server.keystore.jks -keysize 2048 -validity $VALIDITY -storepass $PASSWORD

# Generate cert signing request
echo ""
echo "##############"
echo "Will generate server's cert signing request"
echo "##############"
read -n1 -rsp $'Press any key to continue or Ctrl+C to exit...\n'
keytool -keystore $SERVER_HOSTNAME-server.keystore.jks -alias $SERVER_HOSTNAME -certreq -file $SERVER_HOSTNAME-server.csr  -storepass $PASSWORD

# Sign server certificate with self-signed CA from Step 1
echo ""
echo "##############"
echo "Will sign server certificate"
echo "##############"
read -n1 -rsp $'Press any key to continue or Ctrl+C to exit...\n'
openssl x509 -req -CA rootca.cert.pem -CAkey rootca.key.pem -in $SERVER_HOSTNAME-server.csr -out $SERVER_HOSTNAME-server.cert.pem -days 365 -CAcreateserial -passin pass:$PASSWORD

# Import RootCA cert from step 1 into server's keystore
echo ""
echo "##############"
echo "Will import RootCA into server's keystore"
echo "##############"
read -n1 -rsp $'Press any key to continue or Ctrl+C to exit...\n'
keytool -keystore $SERVER_HOSTNAME-server.keystore.jks -alias CARoot -import -file rootca.cert.pem -storepass $PASSWORD


# Import signed server cert into server's keystore
echo ""
echo "##############"
echo "Will import signed server cert into server's keystore"
echo "##############"
read -n1 -rsp $'Press any key to continue or Ctrl+C to exit...\n'
keytool -keystore $SERVER_HOSTNAME-server.keystore.jks -alias $SERVER_HOSTNAME -import -file $SERVER_HOSTNAME-server.cert.pem -storepass $PASSWORD

# Import RootCA cert into server truststore
echo ""
echo "##############"
echo "Will import RootCA cert into server's truststore"
echo "##############"
read -n1 -rsp $'Press any key to continue or Ctrl+C to exit...\n'
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file rootca.cert.pem -storepass $PASSWORD

### Create client cert
# Create client RSA private key and certificate request
echo ""
echo "##############"
echo "Will create client RSA private key and certificate request"
echo "##############"
read -n1 -rsp $'Press any key to continue or Ctrl+C to exit...\n'
openssl req -nodes -new -keyout mytestclient.key.pem -out mytestclient.csr -days $VALIDITY

# Sign client certificate with CA cert from step 1
echo ""
echo "##############"
echo "Will sign client certificate with CA cert and key from step 1"
echo "##############"
read -n1 -rsp $'Press any key to continue or Ctrl+C to exit...\n'
openssl x509 -req -CA rootca.cert.pem -CAkey rootca.key.pem -in mytestclient.csr -out mytestclient.cert.pem -days 365 -CAcreateserial

echo ""
echo "##############"
echo "Following files were generated"
echo "Server java keystore: $SERVER_HOSTNAME-server.keystore.jks"
echo "Server java truststore: kafka.server.truststore.jks"
echo "Signed Client cert: mytestclient.cert.pem"
echo "Client RSA private key: mytestclient.key.pem"
echo "Client PEM truststore: rootca.cert.pem"

The Kafka config
ssl.keystore.location=/home/kafka/cert-test2/vm-kafka-001-server.keystore.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.truststore.location=/home/kafka/cert-test2/kafka.server.truststore.jks
ssl.truststore.password=password
ssl.client.auth=required
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type=JKS
ssl.truststore.type=JKS

Filebeat config
kafka:
hosts: ["vm-kafka-001.mydomain.com:9093"]
tls:
certificate: /fs/opt/filebeat/config/cert-test2/mytestclient.cert.pem
certificate_key: /fs/opt/filebeat/config/cert-test2/mytestclient.key.pem
certificate_authorities: /fs/opt/filebeat/config/cert-test2/rootca.cert.pem

Hope this helps.

Tsury · September 4, 2016, 12:25pm

EDIT: Before you read all of this, I just wanted to update that I managed to make it work (without the client authentication so far). Apparently if you create the certificate and leave some default fields (not sure which), it will not work! I create a certificate and gave a value to all of the fields and now it works. Now I will try to make it work with client authentication.

Thank you so much steffens and anefassa for your help.

I am making progress but still not there. Trying to get just the server certificate to work for now, without the client authentication.

To make sure we are on the same page I will describe my current system:

DOCKER_HOST_A has a container called DOCKER_A_KAFKA
DOCKER_HOST_B has a container called DOCKER_B_FILEBEAT

These 2 docker hosts sit on different physical machines and networks.

I used the script anefassa supplied on DOCKER_HOST_A to generate all of the relevant files and set a volume on the docker so that DOCKER_A_KAFKA can access them.

At first I got the hostname wrong and got an appropriate message from Filebeat.
I then recreated the files, now with the proper hostname.

This is the config on the Kafka:

ssl.keystore.location=/ssl/MYHOSTNAME-server.keystore.jks
ssl.keystore.password=test1234
ssl.password=test1234

This is the output of an attempt to test to connection from DOCKER_HOST_B to the Kafka server:

openssl s_client -connect MYHOSTNAME:9092 -CAfile ~/Docker/ssl/rootca.cert.pem

CONNECTED(00000003)
depth=1 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = MYHOSTNAME
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
 0 s:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=MYHOSTNAME
   i:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=MYHOSTNAME
 1 s:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=MYHOSTNAME
   i:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=MYHOSTNAME
---
Server certificate
-----BEGIN CERTIFICATE-----
.....
-----END CERTIFICATE-----
subject=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=MYHOSTNAME
issuer=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=MYHOSTNAME
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2364 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 57CC10D01844A36A770BEA6EBD2C675C64E962B63D70549F11135368E2FC2E28
    Session-ID-ctx:
    Master-Key: 1AAE79B8702480B00147E5BF0F8567F9C06167C7D4EA93CAA38E1ECE282CC8E5CF61C5B7DD1DFF79C38AA617A9979337
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1472991439
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
DONE

echo $?

0

Filebeat config:

  tls:
    certificate_authorities:
    - "/ssl/rootca.cert.pem"

Filebeat error:

filebeat_1     | 2016/09/04 12:21:26.751261 log.go:12: WARN Failed to connect to broker MYHOSTNAME:9092: x509: certificate signd by unknown authority
filebeat_1     | 2016/09/04 12:21:26.751384 log.go:16: WARN kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA x509.UnknownAthorityError=x509: certificate signed by unknown authority)

What am I missing here? Is it because the certificate it self signed?
Am I doing something wrong in the creation of the files or the settings?

system · September 22, 2016, 1:05pm

This topic was automatically closed after 21 days. New replies are no longer allowed.