Error Return code 21: SSL encryption between filebeat 5.2 and kafka 1.0 (self signed)


(Gangadhar Mahadevan) #10

Hi Badger, I am getting this error now

2018-02-09T19:38:25Z INFO Setup Beat: filebeat; Version: 5.2.1
2018-02-09T19:38:25Z CRIT Failed loading client certificate%!(EXTRA *errors.errorString=tls: private key does not match public key)
2018-02-09T19:38:25Z ERR failed to initialize kafka plugin as output: 1 error: tls: private key does not match public key
2018-02-09T19:38:25Z CRIT Exiting: error initializing publisher: 1 error: tls: private key does not match public key

I converted the ca-cert file in kafka to pem file using openssl command and copied it to filebeat node.
Also I had a filebeat private key earlier(which I assume is of no use now). Anyway tried both these configs and got the above error

Filebeat config

output.kafka:
  hosts: ["<kafka IP>:9093"]
  topic: '%{[type]}'
  # ssl crt/key files
  ssl.certificate: "/etc/filebeat_ssl/kafka-filebeat.pem"
  ssl.key: "/etc/filebeat_ssl/kafka-key.key" # copied kafka key(ca-key)
  ssl.verification_mode:none
  compression: gzip

Config 2

output.kafka:
  hosts: ["<kafka IP>:9093"]
  topic: '%{[type]}'
  # ssl crt/key files
  ssl.certificate: "/etc/filebeat_ssl/kafka-filebeat.pem"
  ssl.key: "/etc/filebeat_ssl/filebeat.key" # filebeat key generated using openssl genrsa -out filebeat_.key 2048 
  ssl.verification_mode:none
  compression: gzip

This is the files I currently have in my kafka node

ca-cert ca-cert.srl ca-key cert-file cert-signed kafka-filebeat.pem kafka.client.truststore.jks kafka.server.keystore.jks

Thanks so much for your guidance.

Respectfully,
Gangadhar


#11

On the filebeat side you do not want to have ssl.certificate configured unless you are trying to use client authentication. If you just want to encrypt the connection then all you need is ssl.certificate_authorities, which should point to your ca cert in pem format.


(Gangadhar Mahadevan) #12

Ok, it doesn't show the same error now but hitting different one now

2018-02-09T20:23:36Z WARN client/metadata fetching metadata for all topics from broker <kafka IP>:9093

2018-02-09T20:23:36Z WARN Failed to connect to broker <kafka IP>:9093: tls: first record does not look like a TLS handshake

2018-02-09T20:23:36Z WARN kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA tls.RecordHeaderError=tls: first record does not look like a TLS handshake)
2018-02-09T20:23:36Z WARN kafka message: client/metadata no available broker to send metadata request to
2018-02-09T20:23:36Z WARN client/brokers resurrecting 1 dead seed brokers
2018-02-09T20:23:36Z WARN kafka message: Closing Client
2018-02-09T20:23:36Z ERR Kafka connect fails with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
2018-02-09T20:23:36Z ERR Connect failed with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
2018-02-09T20:23:40Z WARN kafka message: Initializing new client
2018-02-09T20:23:40Z WARN client/metadata fetching metadata for all topics from broker <kafka IP>:9093

Updated filebeat config

output.kafka:
  hosts: ["<kafka IP>:9093"]
  topic: '%{[type]}'
  # ssl crt/key files
  # ssl.certificate: "/etc/filebeat_ssl/kafka-filebeat.pem"
  # ssl.key: "/etc/filebeat_ssl/kafka-key.key"
  ssl.certificate_authorities: ["/etc/filebeat_ssl/kafka-filebeat.pem"] # pem format of kafka ca-cert file
  # ssl.verification_mode: none
  compression: gzip

I created the kafka-filebeat.pem file using command

openssl x509 -in ca-cert -out kafka-filebeat.pem -outform PEM
on kafka node and manually copied to filebeat node


#13

What does "openssl s_client -connect :9093 < /dev/null" produce? I would expect it say CONNECTED, followed by some errors in the chain verification, followed by Certificate Chain, Server Certificate, and then the details of the negotiated session.


(Gangadhar Mahadevan) #14

Hi badger, really appreciate your guidance and prompt response. Executed the command from filebeat node. This is the output

    openssl s_client -connect <kafka IP>:9093 < /dev/null
    CONNECTED(00000003)
    depth=1 C = US, ST = Texas, L = Dallas, O = IBM Bluemix Infrastructure, OU = SoftLayer, emailAddress = monitoring@softlayer.com, CN = kafka
    verify error:num=19:self signed certificate in certificate chain
    verify return:0
    ---
    Certificate chain
     0 s:/C=US/ST=Texas/L=Dallas/O=IBM/OU=Softlayer/CN=Gangadhar Mahadevan
       i:/C=US/ST=Texas/L=Dallas/O=IBM Bluemix Infrastructure/OU=SoftLayer/emailAddress=monitoring@softlayer.com/CN=kafka
     1 s:/C=US/ST=Texas/L=Dallas/O=IBM Bluemix Infrastructure/OU=SoftLayer/emailAddress=monitoring@softlayer.com/CN=kafka
       i:/C=US/ST=Texas/L=Dallas/O=IBM Bluemix Infrastructure/OU=SoftLayer/emailAddress=monitoring@softlayer.com/CN=kafka
    ---
    Server certificate
    -----BEGIN CERTIFICATE-----
 certificate
    -----END CERTIFICATE-----
    subject=/C=US/ST=Texas/L=Dallas/O=IBM/OU=Softlayer/CN=Gangadhar Mahadevan
    issuer=/C=US/ST=Texas/L=Dallas/O=IBM Bluemix Infrastructure/OU=SoftLayer/emailAddress=monitoring@softlayer.com/CN=kafka
    ---
    No client certificate CA names sent
    ---
    SSL handshake has read 2494 bytes and written 479 bytes
    ---
    New, TLSv1/SSLv3, Cipher is DHE-DSS-AES128-GCM-SHA256
    Server public key is 1024 bit
    Secure Renegotiation IS supported
    Compression: NONE
    Expansion: NONE
    SSL-Session:
        Protocol  : TLSv1.2
        Cipher    : DHE-DSS-AES128-GCM-SHA256
        Session-ID: 5A7E0AD98AA394F1FDF7051D715AAE1787CF372CD606DB2B31026B093ACDEDE4
        Session-ID-ctx:
        Master-Key: C50EBBAACC596939CCABCDE5D69D6FF255E85A801AB8F318AA649D6B0F4C47BCF19B1010EB212518EC0DDF68AB259BEA
        Key-Arg   : None
        PSK identity: None
        PSK identity hint: None
        SRP username: None
        Start Time: 1518209751
        Timeout   : 300 (sec)
        Verify return code: 19 (self signed certificate in certificate chain)
    ---
    DONE

#15

OK, so openssl connects OK, which suggests kafka is correctly configured. Running out of ideas here. Searching throws up a number of other occurrences of this (e.g. this, this, and this).

Can you remove the non-SSL entry from advertised listeners?


(Andrew Kroh) #16

It would be interesting to see what Wireshark or tcpdump says about the handshake. Maybe you can do a capture like tcpdump -i eth0 -w handshake.pcap tcp port 9093. Then take a look at the pcap in Wireshark.


(Gangadhar Mahadevan) #17

Hi Badger, This is my current kafka config

listeners=ONSL://<IP>:9092,INSL://<IP>:9093


# advertised listener part
advertised.listeners=ONSL://<IP>,INSL://<IP>:9093
inter.broker.listener.name=INSL
listener.security.protocol.map=INSL:SSL,ONSL:PLAINTEXT

When I changed the config to 

listeners=ONSL://<IP>:9092,INSL://IP:9093


# advertised listener part
advertised.listeners=INSL://<IP>:9093
# inter.broker.listener.name=INSL
# listener.security.protocol.map=INSL:SSL,ONSL:PLAINTEXT

and start kafka it fails with error

[2018-02-09 15:29:59,906] FATAL  (kafka.Kafka$)
java.lang.IllegalArgumentException: Error creating broker listeners from 'ONSL://172.16.0.110:9092,INSL://172.16.0.110:9093': No security protocol defined for listener ONSL
        at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:278)

Have I misunderstood anything?


(Gangadhar Mahadevan) #18

Hi Andrew, Do you want to run the tcpdump command on filebeat node or kafka. Both reside at different nodes in my environment. Also should I replace any arguments with values like eth0 etc. Kindly advise


#19

You cannot comment that out. You still have a listener ONSL://<<IP> defined, so you need that map line to let kafka know that it is really PLAINTEXT. Just update the advertised.listeners so that it only includes the INSL


(Gangadhar Mahadevan) #20

Hi,

So the updated config is

# advertised listener part
advertised.listeners=INSL://<IP>:9093
#advertised.listeners=ONSL://<IP>:9092,INSL://<IP>:9093
inter.broker.listener.name=INSL
listener.security.protocol.map=INSL:SSL,ONSL:PLAINTEXT

and kafka logs show

[2018-02-09 15:42:33,373] WARN Failed to send SSL Close message  (org.apache.kafka.common.network.SslTransportLayer)
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:212)
        at org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:170)
        at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:703)
        at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:61)
        at org.apache.kafka.common.network.Selector.doClose(Selector.java:717)
        at org.apache.kafka.common.network.Selector.close(Selector.java:708)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:500)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:398)
        at kafka.network.Processor.poll(SocketServer.scala:535)
        at kafka.network.Processor.run(SocketServer.scala:452)
        at java.lang.Thread.run(Thread.java:745)
[2018-02-09 15:42:35,635] ERROR [KafkaApi-1] Error when handling request {topics=[local-dev-vergil-adc01-rg-syslog]} (kafka.server.KafkaApis)
kafka.common.BrokerEndPointNotAvailableException: Broker `1` does not have listener with name `ListenerName(ONSL)`

this kafka topic local-dev-vergil-adc01-rg-syslog is outside of our filebeat context and seems this topic is affected with above config. Please advise


#21

So something else where requires that you have the ONSL listener. So three questions...

  1. Did kafka start? I think it did.
  2. What did filebeat do once kafka came up?
  3. Can you list both listeners with INSL first? That may break other stuff, but could get us to a good understanding of what the problem is

(Gangadhar Mahadevan) #22

Hi,

Did kafka start? I think it did. - Yes
What did filebeat do once kafka came up? - tls: first record does not look like a TLS handshake
Can you list both listeners with INSL first? That may break other stuff, but could get us to a good understanding of what the problem is

Config

# advertised listener part
#advertised.listeners=INSL://<IP>:9093
advertised.listeners=INSL://<IP>:9092,INSL://<IP>:9093
inter.broker.listener.name=INSL
listener.security.protocol.map=INSL:SSL,ONSL:PLAINTEXT

Error

[2018-02-09 16:06:05,668] FATAL  (kafka.Kafka$)
java.lang.IllegalArgumentException: requirement failed: Each listener must have a different name, listeners: INSL://172.16.0.110:9092,INSL://172.16.0.110:9093

and kafka process stopped


#23

Try

advertised.listeners=INSL://<IP>:9093,ONSL://<IP>:9092

(Gangadhar Mahadevan) #24

Tried that, kafka didn't throw any error but filebeat has same tls: first record does not look like a TLS handshake

Also when I generated the kafka cert I had my ssl config as

cat > cert_info_kafka << EOF
[req]
default_bits = 2048
prompt = no
default_md = sha512
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C=US
ST=xx
L=xx
O=xxx
OU=xx
emailAddress=xx@xx.com
CN = kafka

[ req_ext ]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

[ alt_names ]
IP.1 = <kafka IP>

[ usr_cert ]
# Extensions for server certificates.
basicConstraints = **CA:FALSE**
nsCertType = client, server
nsComment = "OpenSSL Kafka Server / Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = **critical, digitalSignature, keyEncipherment, keyAgreement, nonRepudiation**
extendedKeyUsage = serverAuth, clientAuth

EOF

but shouldn't CA be true instead. Am thinking if I made any mistake during cert generation. Any thoughts? I was hoping to use SAN so I can list all kafka IP while generating cert and use that one cert across cluster


#25

I know nothing about generating certificates, our corporate CA handles that for me, but in post 6 of this thread Steffens provided links to documentation. Before you worry about SANs, just generate a name-matched cert for your kafka server, exactly following the documented steps. Get the most basic use-case working and worry about extras afterwards.


(Gangadhar Mahadevan) #26

Thanks so much Badger and everyone. I will check more details and update the thread with my further findings


(Gangadhar Mahadevan) #27

Hi Badger, Steffen, Andrew

I made some progress over weekened, apparently it looked like I messed up something while generating my self-signed CA. I am able to send metrics from filebeat to kafka with ssl enabled on both kafka and filebeat end with ssl.verification_mode: none, but however when I comment that setting I see a different issue

Working config

Filebeat

Kafka output

output.kafka:
  hosts: ["<kafka IP>:9093"]
  topic: '%{[type]}'
  ssl.certificate_authorities: ["/etc/kafka-fb.pem"]
  ssl.verification_mode: none
  compression: gzip

Kafka config

listeners=ONSL://<kafka IP>:9092,INSL://<kafka IP>:9093


# advertised listener part
advertised.listeners=ONSL://<kafka IP>:9092,INSL://<kafka IP>:9093
inter.broker.listener.name=INSL
listener.security.protocol.map=INSL:SSL,ONSL:PLAINTEXT


# ssl encryption config from docs
#ssl.keystore.location=/opt/obuildfactory/jdk-1.8.0-openjdk-x86_64/jre/bin/kafka_certs/server.keystore.jks
#ssl.keystore.password=<password>
ssl.key.password=<password>
ssl.truststore.location=/opt/obuildfactory/jdk-1.8.0-openjdk-x86_64/jre/bin/kafka_certs/client.truststore.jks
ssl.truststore.password=<password>

From the docs, ssl.verification_mode: none setting is not advisable and when I don't use that in filebeat config the logs show

2018-02-12T17:40:31Z WARN kafka message: Initializing new client
2018-02-12T17:40:31Z WARN client/metadata fetching metadata for all topics from broker 172.16.0.110:9093

2018-02-12T17:40:32Z WARN Failed to connect to broker <kafka IP>:9093: x509: cannot validate certificate for 172.16.0.110 because it doesn't contain any IP SANs

2018-02-12T17:40:32Z WARN kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA x509.HostnameError=x509: cannot validate certificate for <kafka IP>because it doesn't contain any IP SANs)
2018-02-12T17:40:32Z WARN kafka message: client/metadata no available broker to send metadata request to
2018-02-12T17:40:32Z WARN client/brokers resurrecting 1 dead seed brokers
2018-02-12T17:40:32Z WARN client/metadata retrying after 250ms... (3 attempts remaining)

2018-02-12T17:40:32Z WARN client/metadata fetching metadata for all topics from broker <kafka IP>:9093

I use IP's because we don't have DNS resolution in our pre-prod environments and in my dev setup which I am trying things out. So I was using IP SAN's while generating certificate. Is there something that can be done to get it working without ssl.verification_mode: none set. Please advise.

Regards,
Gangadhar


#28

Go through the certificate generation process again, but this time include the IP SANs.


(Gangadhar Mahadevan) #29

I did it the previous time, however when I check the pem file contents it doesn't have any SAN details. Let me re-try generating CA. Thanks!