# Error Return code 21: SSL encryption between filebeat 5.2 and kafka 1.0 (self signed)

**URL:** <https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 8, 2018, 9:13pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126 "2018-02-08T21:13:36Z")\
**Posts on this page:** 11\
**Page:** 2

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 9, 2018, 9:51pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126/21 "2018-02-09T21:51:46Z")

</div>

So something else where requires that you have the ONSL listener. So three questions...

1. Did kafka start? I think it did.
2. What did filebeat do once kafka came up?
3. Can you list both listeners with INSL first? That may break other stuff, but could get us to a good understanding of what the problem is

---

<div class="post-metadata">

**Author:** ![Gangadhar\_Mahadevan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gangadhar_mahadevan/32/62579_2.png) [@Gangadhar\_Mahadevan](https://discuss.elastic.co/u/Gangadhar_Mahadevan)\
**Post date:** [February 9, 2018, 10:08pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126/22 "2018-02-09T22:08:12Z")

</div>

Hi,

Did kafka start? I think it did. - Yes  
What did filebeat do once kafka came up? - tls: first record does not look like a TLS handshake  
Can you list both listeners with INSL first? That may break other stuff, but could get us to a good understanding of what the problem is

Config

```
# advertised listener part
#advertised.listeners=INSL://<IP>:9093
advertised.listeners=INSL://<IP>:9092,INSL://<IP>:9093
inter.broker.listener.name=INSL
listener.security.protocol.map=INSL:SSL,ONSL:PLAINTEXT

```

Error

```
[2018-02-09 16:06:05,668] FATAL (kafka.Kafka$)
java.lang.IllegalArgumentException: requirement failed: Each listener must have a different name, listeners: INSL://172.16.0.110:9092,INSL://172.16.0.110:9093

```

and kafka process stopped

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 9, 2018, 10:11pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126/23 "2018-02-09T22:11:08Z")

</div>

Try

```
advertised.listeners=INSL://<IP>:9093,ONSL://<IP>:9092
```

---

<div class="post-metadata">

**Author:** ![Gangadhar\_Mahadevan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gangadhar_mahadevan/32/62579_2.png) [@Gangadhar\_Mahadevan](https://discuss.elastic.co/u/Gangadhar_Mahadevan)\
**Post date:** [February 9, 2018, 10:20pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126/24 "2018-02-09T22:20:24Z")

</div>

Tried that, kafka didn't throw any error but filebeat has same `tls: first record does not look like a TLS handshake`

Also when I generated the kafka cert I had my ssl config as

```
cat > cert_info_kafka << EOF
[req]
default_bits = 2048
prompt = no
default_md = sha512
req_extensions = req_ext
distinguished_name = dn

[dn]
C=US
ST=xx
L=xx
O=xxx
OU=xx
emailAddress=xx@xx.com
CN = kafka

[req_ext]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

[alt_names]
IP.1 = <kafka IP>

[usr_cert]
# Extensions for server certificates.
basicConstraints = **CA:FALSE**
nsCertType = client, server
nsComment = "OpenSSL Kafka Server / Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = **critical, digitalSignature, keyEncipherment, keyAgreement, nonRepudiation**
extendedKeyUsage = serverAuth, clientAuth

EOF

```

but shouldn't CA be true instead. Am thinking if I made any mistake during cert generation. Any thoughts? I was hoping to use SAN so I can list all kafka IP while generating cert and use that one cert across cluster

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 9, 2018, 10:28pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126/25 "2018-02-09T22:28:02Z")

</div>

I know nothing about generating certificates, our corporate CA handles that for me, but in post 6 of this thread Steffens provided links to documentation. Before you worry about SANs, just generate a name-matched cert for your kafka server, exactly following the documented steps. Get the most basic use-case working and worry about extras afterwards.

---

<div class="post-metadata">

**Author:** ![Gangadhar\_Mahadevan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gangadhar_mahadevan/32/62579_2.png) [@Gangadhar\_Mahadevan](https://discuss.elastic.co/u/Gangadhar_Mahadevan)\
**Post date:** [February 9, 2018, 10:29pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126/26 "2018-02-09T22:29:30Z")

</div>

Thanks so much Badger and everyone. I will check more details and update the thread with my further findings

---

<div class="post-metadata">

**Author:** ![Gangadhar\_Mahadevan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gangadhar_mahadevan/32/62579_2.png) [@Gangadhar\_Mahadevan](https://discuss.elastic.co/u/Gangadhar_Mahadevan)\
**Post date:** [February 12, 2018, 6:12pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126/27 "2018-02-12T18:12:26Z")

</div>

Hi Badger, Steffen, Andrew

I made some progress over weekened, apparently it looked like I messed up something while generating my self-signed CA. I am able to send metrics from filebeat to kafka with ssl enabled on both kafka and filebeat end with ssl.verification\_mode: none, but however when I comment that setting I see a different issue

Working config

Filebeat

### Kafka output

```
output.kafka:
  hosts: ["<kafka IP>:9093"]
  topic: '%{[type]}'
  ssl.certificate_authorities: ["/etc/kafka-fb.pem"]
  ssl.verification_mode: none
  compression: gzip

```

Kafka config

```
listeners=ONSL://<kafka IP>:9092,INSL://<kafka IP>:9093

# advertised listener part
advertised.listeners=ONSL://<kafka IP>:9092,INSL://<kafka IP>:9093
inter.broker.listener.name=INSL
listener.security.protocol.map=INSL:SSL,ONSL:PLAINTEXT

# ssl encryption config from docs
#ssl.keystore.location=/opt/obuildfactory/jdk-1.8.0-openjdk-x86_64/jre/bin/kafka_certs/server.keystore.jks
#ssl.keystore.password=<password>
ssl.key.password=<password>
ssl.truststore.location=/opt/obuildfactory/jdk-1.8.0-openjdk-x86_64/jre/bin/kafka_certs/client.truststore.jks
ssl.truststore.password=<password>

```

From the docs, ssl.verification\_mode: none setting is not advisable and when I don't use that in filebeat config the logs show

```
2018-02-12T17:40:31Z WARN kafka message: Initializing new client
2018-02-12T17:40:31Z WARN client/metadata fetching metadata for all topics from broker 172.16.0.110:9093

2018-02-12T17:40:32Z WARN Failed to connect to broker <kafka IP>:9093: x509: cannot validate certificate for 172.16.0.110 because it doesn't contain any IP SANs

2018-02-12T17:40:32Z WARN kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA x509.HostnameError=x509: cannot validate certificate for <kafka IP>because it doesn't contain any IP SANs)
2018-02-12T17:40:32Z WARN kafka message: client/metadata no available broker to send metadata request to
2018-02-12T17:40:32Z WARN client/brokers resurrecting 1 dead seed brokers
2018-02-12T17:40:32Z WARN client/metadata retrying after 250ms... (3 attempts remaining)

2018-02-12T17:40:32Z WARN client/metadata fetching metadata for all topics from broker <kafka IP>:9093

```

I use IP's because we don't have DNS resolution in our pre-prod environments and in my dev setup which I am trying things out. So I was using IP SAN's while generating certificate. Is there something that can be done to get it working without ssl.verification\_mode: none set. Please advise.

Regards,  
Gangadhar

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 12, 2018, 6:47pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126/28 "2018-02-12T18:47:42Z")

</div>

> [@Gangadhar\_Mahadevan](#):
>
> Is there something that can be done to get it working without ssl.verification\_mode: none set.

Go through the certificate generation process again, but this time include the IP SANs.

---

<div class="post-metadata">

**Author:** ![Gangadhar\_Mahadevan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gangadhar_mahadevan/32/62579_2.png) [@Gangadhar\_Mahadevan](https://discuss.elastic.co/u/Gangadhar_Mahadevan)\
**Post date:** [February 12, 2018, 6:49pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126/29 "2018-02-12T18:49:55Z")

</div>

I did it the previous time, however when I check the pem file contents it doesn't have any SAN details. Let me re-try generating CA. Thanks!

---

<div class="post-metadata">

**Author:** ![Gangadhar\_Mahadevan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gangadhar_mahadevan/32/62579_2.png) [@Gangadhar\_Mahadevan](https://discuss.elastic.co/u/Gangadhar_Mahadevan)\
**Post date:** [February 12, 2018, 7:41pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126/30 "2018-02-12T19:41:34Z")

</div>

Hi Badger,

I re-generated a kafka CA and kafka keystore,truststore needed and converted the cert to pem file to use it in filebeat. Still no luck.

Below is the process I used to generate CA. Please let me know if I missed something

CA config file  
cat \> updated\_cert\_info\_kafka \<\< EOF  
[req]  
default\_bits = 2048  
prompt = no  
default\_md = sha512  
req\_extensions = req\_ext  
distinguished\_name = dn

```
[dn]
C=US
ST=Texas
L=Dallas
O=<val>
OU=<val>
emailAddress=<val>
CN = kafka

[req_ext]
basicConstraints = CA:TRUE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyCertSign, cRLSign
subjectAltName = @alt_names

[alt_names]
IP.1 = <kafka IP>

EOF

```

From kafka docs

```
keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA

keytool -list -v -keystore server.keystore.jks

openssl req -new -x509 -keyout ca-key -out ca-cert -days 365 -config updated_cert_info_kafka

keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert

keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:{ca-password}

keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed
openssl x509 -in ca-cert -out mycert.pem -outform PEM

```

This is the certificate details

openssl x509 -text -noout -in

```
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            <serial number>
    Signature Algorithm: sha512WithRSAEncryption
        Issuer: C=US, ST=Texas, L=Dallas, O=<val>, OU=<val>/emailAddress=<val>, CN=kafka
        Validity
            Not Before: Feb 9 23:21:38 2018 GMT
            Not After : Feb 9 23:21:38 2019 GMT
        Subject: C=US, ST=Texas, L=Dallas, O=<val>, OU=<val>/emailAddress=<val>, CN=kafka
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    <key>
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha512WithRSAEncryption
        <key>

```

Filebeat error

```
2018-02-12T19:31:24Z WARN Failed to connect to broker 172.16.0.110:9093: x509: cannot validate certificate for 172.16.0.110 because it doesn't contain any IP SANs

2018-02-12T19:31:24Z WARN kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA x509.HostnameError=x509: cannot validate certificate for 172.16.0.110 because it doesn't contain any IP SANs)

```

Kindly advise

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 12, 2018, 7:41pm UTC](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126/31 "2018-03-12T19:41:36Z")

</div>

Closed on March 12, 2018, 7:41pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/error-return-code-21-ssl-encryption-between-filebeat-5-2-and-kafka-1-0-self-signed/119126.md?page=1)