# TLS for Filebeat Kafka Output

**URL:** <https://discuss.elastic.co/t/tls-for-filebeat-kafka-output/58756>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 23, 2016, 11:35pm UTC](https://discuss.elastic.co/t/tls-for-filebeat-kafka-output/58756 "2016-08-23T23:35:51Z")\
**Posts on this page:** 1\
**Showing post:** 5

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 28, 2016, 3:45pm UTC](https://discuss.elastic.co/t/tls-for-filebeat-kafka-output/58756/5 "2016-08-28T15:45:51Z")

</div>

I just did try with client authentication without any problems.

In my setup I've got a `root ca` (named: `ca`) and an `intermediate ca` (named `sign-ca`) signing actual certificates used by client and server.

For beats I'm using PEM format and for kafka jks based key store.

The trustchain for beats is build by executing (watch order):

```auto
	$ cat certs/sign-ca.cert.pem certs/ca.cert.pem > certs/trustchain.cert.pem

```

The trustchain for kafka is build by running:

```auto
	# add ca certificate to trustchain.jks
	$ keytool -importcert -file certs/ca.cert.pem -alias ca -noprompt -storepass "${PASS}" -keystore certs/trustchain.jks
	# add sign-ca certificate to certs/trustchain.jks
	$ keytool -importcert -file certs/sign-ca.cert.pem -alias 'sign-ca' -noprompt -storepass "${PASS}" -keystore certs/trustchain.jks

```

The `trustchain.(cert.pem/jks)` is required for validation. That's why these files do only contain the public certificates of the root CA and the intermediate CA.

Having server and client certificates (both signed by `sign-ca`) + private keys, next I configure kafka to enable SSL server certificate (no client authentication yet):

kafka server conf:

```auto
listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
ssl.keystore.location=./ssl/private/localhost.jks
ssl.keystore.password=password
ssl.key.password=password

```

The file `localhost.jks` contains the server certificate + private key only imported from PEM files (generated with openssl) using keytool:

```auto
	# create pkcs12 store
	$ openssl pkcs12 -export -out private/localhost.pkcs12 -in certs/localhost.cert.pem -inkey private/localhost.key.pem -passout "pass:password"
	# import server key into jks keystore
	$ echo "password" | keytool -importkeystore -srckeystore private/localhost.pkcs12 -destkeystore private/localhost.jks -srcstoretype pkcs12 -storepass "password"

```

Starting kafka I check SSL validation working via openssl:

```auto
$ openssl s_client -connect localhost:9093 -tls1_2 -CAfile certs/trustchain.cert.pem

```

if openssl does quit something went wrong.

Next enable client authentication + add `trustchain.jks` for client certificate validation in kafka:

kafka.conf

```auto
listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
ssl.keystore.location=./ssl/private/localhost.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.truststore.location=./ssl/certs/trustchain.jks
ssl.truststore.password=password
ssl.client.auth=required

```

And check once again with openssl:

```auto
$ openssl s_client -connect localhost:9093 -tls1_2 -CAfile certs/trustchain.cert.pem -cert certs/client.localhost.cert.pem -key private/client.localhost.key.pem
$ echo $?

```

Note: the client its private key is unencrypted, hence no passsword. [This PR](https://github.com/elastic/beats/pull/2330) adds support for encrypted private keys.

If server certificate succeeds, but client authentication fails openssl will quit immediately with exit code 1, but no error will be written. Unfortunately kafka didn't log any error for me too. In this case something's wrong with the certificates or your setup.

Next let's configure filebeat:

filebeat.conf:

```auto
output:
  kafka:
    hosts: ["localhost:9093"]
    ...
    ssl:
      certificate: ../ssl/certs/client.localhost.cert.pem
      certificate_key: ../ssl/private/client.localhost.key.pem
      certificate_authorities:
        - ../ssl/certs/trustchain.cert.pem

```

Using this setup, filebeat+kafka do indeed mutual SSL based authentication.

Setting up filebeat without client certificate, kafka output fails with:

```auto
2016/08/28 15:11:12.529914 log.go:12: WARN Failed to connect to broker localhost:9093: local error: tls: no renegotiation
2016/08/28 15:11:12.529939 log.go:16: WARN kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA *net.OpError=local error: tls: no renegotiation)
2016/08/28 15:11:12.529947 log.go:16: WARN kafka message: client/metadata no available broker to send metadata request to

```

---

_[View the full topic](https://discuss.elastic.co/t/tls-for-filebeat-kafka-output/58756)._