Setting Filebeat 5 with Kafka 0.10 over TLS

Elastic Stack Beats
5

EDIT: Before you read all of this, I just wanted to update that I managed to make it work (without the client authentication so far). Apparently if you create the certificate and leave some default fields (not sure which), it will not work! I create a certificate and gave a value to all of the fields and now it works. Now I will try to make it work with client authentication.

Thank you so much steffens and anefassa for your help.

I am making progress but still not there. Trying to get just the server certificate to work for now, without the client authentication.

To make sure we are on the same page I will describe my current system:

DOCKER_HOST_A has a container called DOCKER_A_KAFKA
DOCKER_HOST_B has a container called DOCKER_B_FILEBEAT

These 2 docker hosts sit on different physical machines and networks.

I used the script anefassa supplied on DOCKER_HOST_A to generate all of the relevant files and set a volume on the docker so that DOCKER_A_KAFKA can access them.

At first I got the hostname wrong and got an appropriate message from Filebeat.
I then recreated the files, now with the proper hostname.

This is the config on the Kafka:

ssl.keystore.location=/ssl/MYHOSTNAME-server.keystore.jks
ssl.keystore.password=test1234
ssl.password=test1234

This is the output of an attempt to test to connection from DOCKER_HOST_B to the Kafka server:

openssl s_client -connect MYHOSTNAME:9092 -CAfile ~/Docker/ssl/rootca.cert.pem

CONNECTED(00000003)
depth=1 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = MYHOSTNAME
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
 0 s:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=MYHOSTNAME
   i:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=MYHOSTNAME
 1 s:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=MYHOSTNAME
   i:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=MYHOSTNAME
---
Server certificate
-----BEGIN CERTIFICATE-----
.....
-----END CERTIFICATE-----
subject=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=MYHOSTNAME
issuer=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=MYHOSTNAME
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2364 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 57CC10D01844A36A770BEA6EBD2C675C64E962B63D70549F11135368E2FC2E28
    Session-ID-ctx:
    Master-Key: 1AAE79B8702480B00147E5BF0F8567F9C06167C7D4EA93CAA38E1ECE282CC8E5CF61C5B7DD1DFF79C38AA617A9979337
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1472991439
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
DONE

echo $?

0

Filebeat config:

  tls:
    certificate_authorities:
    - "/ssl/rootca.cert.pem"

Filebeat error:

filebeat_1     | 2016/09/04 12:21:26.751261 log.go:12: WARN Failed to connect to broker MYHOSTNAME:9092: x509: certificate signd by unknown authority
filebeat_1     | 2016/09/04 12:21:26.751384 log.go:16: WARN kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA x509.UnknownAthorityError=x509: certificate signed by unknown authority)

What am I missing here? Is it because the certificate it self signed?
Am I doing something wrong in the creation of the files or the settings?