Logstash 7.5 with SSL giving SSLV3_ALERT_BAD_CERTIFICATE

Elastic Stack Logstash
1

Hi everyone,

I know that there are a couple of topics regarding this problem. I reviewed them but there are for older logstash versions or the solution doesn't work for me.

I have the following logstash configuration:
file logstash/conf.d/logstash-nginx-es.conf

input {
    beats {
        port => 5400
        ssl => true
        ssl_certificate_authorities => ["/etc/elkcerts/ca.crt"]
        ssl_certificate => "/etc/elkcerts/server.crt"
        ssl_key => "/etc/elkcerts/server.key.p8"
        ssl_verify_mode => "force_peer"
    }
}
---
output {
    elasticsearch {
        hosts => ["https://kibana.rebellionpay.com:9201"]
        index => "prodlogs-%{+YYYY.MM.dd}"
        document_type => "nginx_logs"
        user => "XXXXXXXXXXX"
        password => "XXXXXXXXXXX"
    }
    stdout { codec => rubydebug }
}

In logstash.yaml file, in the X-Pack section both monitoring and management are disabled.

In the client side I have a filebeat process with the following configuration:
file filebeat/filebeat.yml

output.logstash:
    hosts: ["kibana.mydomain.com"]

    ssl.certificate_authorities: ["/etc/elkcerts/ca.crt"]

    ssl.certificate: "/etc/elkcerts/client.crt"

    ssl.key: "/etc/elkcerts/client.key"

    ssl.supported_protocols: "TLSv1.2"

And when I run the logstash server using bin/logstash --log.level debug --config.path /etc/logstash/logstash.yaml I get the following error:

[2019-12-19T16:21:40,511][DEBUG][org.logstash.netty.SslSimpleBuilder][main] Available ciphers:[ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES256-GCM-SHA384, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-CHACHA20-POLY1305, ECDHE-RSA-CHACHA20-POLY1305, ECDHE-PSK-CHACHA20-POLY1305, ECDHE-ECDSA-AES128-SHA, ECDHE-ECDSA-AES128-SHA256, ECDHE-RSA-AES128-SHA, ECDHE-RSA-AES128-SHA256, ECDHE-PSK-AES128-CBC-SHA, ECDHE-ECDSA-AES256-SHA, ECDHE-ECDSA-AES256-SHA384, ECDHE-RSA-AES256-SHA, ECDHE-RSA-AES256-SHA384, ECDHE-PSK-AES256-CBC-SHA, AES128-GCM-SHA256, AES256-GCM-SHA384, AES128-SHA, AES128-SHA256, PSK-AES128-CBC-SHA, AES256-SHA, AES256-SHA256, PSK-AES256-CBC-SHA, DES-CBC3-SHA]
[2019-12-19T16:21:40,512][DEBUG][org.logstash.netty.SslSimpleBuilder][main] Ciphers:  [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256]
[2019-12-19T16:21:40,512][DEBUG][org.logstash.netty.SslSimpleBuilder][main] Certificate Authorities: [/etc/elkcerts/ca.crt]
[2019-12-19T16:21:40,513][DEBUG][org.logstash.netty.SslSimpleBuilder][main] Load certificates collection
[2019-12-19T16:21:40,513][DEBUG][org.logstash.netty.SslSimpleBuilder][main] Loading certificates from file /etc/elkcerts/ca.crt
[2019-12-19T16:21:40,518][DEBUG][org.logstash.netty.SslSimpleBuilder][main] TLS: [TLSv1, TLSv1.1, TLSv1.2]
[2019-12-19T16:21:40,523][DEBUG][logstash.codecs.plain    ][main] config LogStash::Codecs::Plain/@id = "plain_0e20a8f9-3d1f-46d5-9e0b-e188aa27b5d0"
[2019-12-19T16:21:40,523][DEBUG][logstash.codecs.plain    ][main] config LogStash::Codecs::Plain/@enable_metric = true
[2019-12-19T16:21:40,524][DEBUG][logstash.codecs.plain    ][main] config LogStash::Codecs::Plain/@charset = "UTF-8"
[2019-12-19T16:21:40,527][DEBUG][io.netty.handler.ssl.ReferenceCountedOpenSslEngine][main] SSL_read failed with 1: OpenSSL error: 268436498 error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
[2019-12-19T16:21:40,531][DEBUG][org.logstash.beats.BeatsHandler][main] [local: 0.0.0.0:5400, remote: 52.211.65.130:57822] Handling exception: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
...

If you know why can this be happening or what can I do to solve it, I'd be so grateful :slight_smile:

Thanks so much everyone.

2

I have this in logstash config:
ssl_verify_mode => "peer"

and I have the followings in beat config also
ssl.enable: true

but I don't have ssl.supported_protocols: "TLSv1.2"

3

This is an SSL alert.
Alerts are sent from one party in the connection to the other party to inform them that they detected a problem. Typically these are fatal, and the first party then terminates the connection.

In this case Logstash is reporting that it received an alert from someone (probably beats, since that's the only input you have) that says "I don't trust your certificate".

Which means the issue here is that beats is not configured to trust the certificate that Logstash provides.
You probably need to modify the beats config to include ss.certificate_authorities

4

Thank you!! I remade the process of generating the certs using the elaticsearch certutil tool and I signed both the client and server certs with the same CA and now it's working!

Thanks again!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.