Cannot setup SSL for Filebeat to Logstash


(Luke Briner) #1

Filebeat version 6.1 for Windows
Logstash version 6.1 on Ubuntu 16:04
Curl running from Ubuntu 16:04

I've followed the instructions https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html to setup SSL after getting a working system without SSL. I don't want to use a client certificate so my filebeat output config is this:

output.logstash:
  # The Logstash hosts
  hosts: ["logstash-1.internal.local:5044"]

(Note, URL domain has been changed for privacy). I have enabled ssl in the logstash beats input like this:

input {
  beats {
    port => "5044"
    ssl => true
    ssl_certificate => "/etc/ssl/certs/wildcard4-combined.cer"
    ssl_key => "/etc/ssl/private/wildcard4.pem"
  }
}

The certificate is a valid cert chained to a trusted root (the help documentation should not assume that you are using self-signed certificates!) but when I try and run curl -v against the host to verify the connection, I get the following error:

* Rebuilt URL to: https://logstash-1.internal.local:5044/
*   Trying 10.1.26.23...
* Connected to logstash-1.internal.local (10.1.26.23) port 5044 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 595 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: *.internal.local (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: OU=Domain Control Validated,OU=PositiveSSL Wildcard,CN=*.internal.local
*        start date: Thu, 14 Jan 2016 00:00:00 GMT
*        expire date: Sat, 13 Apr 2019 23:59:59 GMT
*        issuer: C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Domain Validation Secure Server CA
*        compression: NULL
* ALPN, server did not agree to a protocol
> GET / HTTP/1.1
> Host: logstash-1.internal.local:5044
> User-Agent: curl/7.47.0
> Accept: */*
>
* GnuTLS recv error (-54): Error in the pull function.
* Closing connection 0
curl: (56) GnuTLS recv error (-54): Error in the pull function.

I was initially trying to run logstash behind an nginx proxy and if I run curl -v against nginx, it works fine so the implication from the error is that logstash (or jruby-openssl) doesn't support enough cipher suites and specifically, not what curl is trying to use. I ran a bash script to test supported ciphers suites. nginx supports 13, logstash only supports 4. Since this will only move in one direction, if this is the problem, then logstash needs updating urgently to support newer cipher suites for SSL to be usable.

If I try and run filebeat (from Windows if it's important) with this configuration, I get the following:

2018/01/31 09:39:28.188930 output.go:92: ERR Failed to publish events: client is not connected
2018/01/31 09:39:28.230972 async.go:235: ERR Failed to publish events caused by: lumberjack protocol error
2018/01/31 09:39:28.231973 async.go:235: ERR Failed to publish events caused by: client is not connected

(Pier-Hugues Pellerin) #2

@Luke_Briner This look like a bug, the underlying implementation doesn't use jruby-openssl but bouncy-castle and should support a few ciphers out of the box, the defaults are defined here, They are also configurable.

Would you mind starting Logstash with --log.level debug it will give a few TLS/Ciphers information on the first connection.


(Luke Briner) #3

Thanks for the help. I've run it with debug and the first problem is that the available ciphers in Logstash by default does not match the list in https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-cipher_suites

I added the config to logstash to use the list described in the docs, which DOES include the one that curl is trying to use. It now looks like it should be working (but it isn't) although it is very confusing how the names match but not quite and I'm not sure whether this is the problem. For instance, curl is displaying:

ECDHE_RSA_AES_256_GCM_SHA384 but the logstash mapping that gets outputted is:
ECDHE-RSA-AES256-GCM-SHA384

The available ciphers listed by logstash are:

SslSimpleBuilder - Ciphers: [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256]

and the full mapping, which looks relevant (but which has the slight typo) is:

Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM- SHA384

Filebeat also still errors. When it attempts to send logs, logstash logs "OPENSSL_internal:WRONG_VERSION_NUMBER"


(Pier-Hugues Pellerin) #4

The names are not an exact match but there is internal conversion inside the library to take care of that.

Can you add a bit more of the Logstash logs here?


(Pier-Hugues Pellerin) #5

I mean, Logstash on start should list ciphers and things like that when you run with --log.level debug


(Luke Briner) #6
[DEBUG] 2018-01-31 17:05:06.778 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-ECDSA-CHACHA20-POLY1305
[DEBUG] 2018-01-31 17:05:06.778 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-ECDSA-CHACHA20-POLY1305
[DEBUG] 2018-01-31 17:05:06.778 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-RSA-CHACHA20-POLY1305
[DEBUG] 2018-01-31 17:05:06.779 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-RSA-CHACHA20-POLY1305
[DEBUG] 2018-01-31 17:05:06.779 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-PSK-CHACHA20-POLY1305
[DEBUG] 2018-01-31 17:05:06.779 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_PSK_WITH_CHACHA20_POLY1305_SHA256 => ECDHE-PSK-CHACHA20-POLY1305
[DEBUG] 2018-01-31 17:05:06.780 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 => ECDHE-ECDSA-AES128-GCM-SHA256
[DEBUG] 2018-01-31 17:05:06.780 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 => ECDHE-ECDSA-AES128-GCM-SHA256
[DEBUG] 2018-01-31 17:05:06.780 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 => ECDHE-RSA-AES128-GCM-SHA256
[DEBUG] 2018-01-31 17:05:06.780 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_128_GCM_SHA256 => ECDHE-RSA-AES128-GCM-SHA256
[DEBUG] 2018-01-31 17:05:06.780 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384
[DEBUG] 2018-01-31 17:05:06.781 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 => ECDHE-ECDSA-AES256-GCM-SHA384
[DEBUG] 2018-01-31 17:05:06.781 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
[DEBUG] 2018-01-31 17:05:06.781 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384
[DEBUG] 2018-01-31 17:05:06.781 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA => ECDHE-ECDSA-AES128-SHA
[DEBUG] 2018-01-31 17:05:06.781 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA => ECDHE-ECDSA-AES128-SHA
[DEBUG] 2018-01-31 17:05:06.781 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 => ECDHE-ECDSA-AES128-SHA256
[DEBUG] 2018-01-31 17:05:06.782 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 => ECDHE-ECDSA-AES128-SHA256
[DEBUG] 2018-01-31 17:05:06.782 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA => ECDHE-RSA-AES128-SHA
[DEBUG] 2018-01-31 17:05:06.782 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA => ECDHE-RSA-AES128-SHA
[DEBUG] 2018-01-31 17:05:06.782 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 => ECDHE-RSA-AES128-SHA256
[DEBUG] 2018-01-31 17:05:06.782 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA256 => ECDHE-RSA-AES128-SHA256
[DEBUG] 2018-01-31 17:05:06.783 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_PSK_WITH_AES_128_CBC_SHA => ECDHE-PSK-AES128-CBC-SHA
[DEBUG] 2018-01-31 17:05:06.783 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_PSK_WITH_AES_128_CBC_SHA => ECDHE-PSK-AES128-CBC-SHA
[DEBUG] 2018-01-31 17:05:06.783 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA => ECDHE-ECDSA-AES256-SHA
[DEBUG] 2018-01-31 17:05:06.783 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA => ECDHE-ECDSA-AES256-SHA
[DEBUG] 2018-01-31 17:05:06.783 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 => ECDHE-ECDSA-AES256-SHA384
[DEBUG] 2018-01-31 17:05:06.784 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 => ECDHE-ECDSA-AES256-SHA384
[DEBUG] 2018-01-31 17:05:06.784 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA => ECDHE-RSA-AES256-SHA
[DEBUG] 2018-01-31 17:05:06.784 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA => ECDHE-RSA-AES256-SHA
[DEBUG] 2018-01-31 17:05:06.784 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 => ECDHE-RSA-AES256-SHA384
[DEBUG] 2018-01-31 17:05:06.784 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA384 => ECDHE-RSA-AES256-SHA384
[DEBUG] 2018-01-31 17:05:06.785 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_ECDHE_PSK_WITH_AES_256_CBC_SHA => ECDHE-PSK-AES256-CBC-SHA
[DEBUG] 2018-01-31 17:05:06.785 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_ECDHE_PSK_WITH_AES_256_CBC_SHA => ECDHE-PSK-AES256-CBC-SHA
[DEBUG] 2018-01-31 17:05:06.785 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_RSA_WITH_AES_128_GCM_SHA256 => AES128-GCM-SHA256
[DEBUG] 2018-01-31 17:05:06.785 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_RSA_WITH_AES_128_GCM_SHA256 => AES128-GCM-SHA256
[DEBUG] 2018-01-31 17:05:06.785 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_RSA_WITH_AES_256_GCM_SHA384 => AES256-GCM-SHA384
[DEBUG] 2018-01-31 17:05:06.786 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_RSA_WITH_AES_256_GCM_SHA384 => AES256-GCM-SHA384
[DEBUG] 2018-01-31 17:05:06.786 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_RSA_WITH_AES_128_CBC_SHA => AES128-SHA

(Luke Briner) #7
[DEBUG] 2018-01-31 17:05:06.786 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_RSA_WITH_AES_128_CBC_SHA => AES128-SHA
[DEBUG] 2018-01-31 17:05:06.786 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_RSA_WITH_AES_128_CBC_SHA256 => AES128-SHA256
[DEBUG] 2018-01-31 17:05:06.786 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_RSA_WITH_AES_128_CBC_SHA256 => AES128-SHA256
[DEBUG] 2018-01-31 17:05:06.787 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_PSK_WITH_AES_128_CBC_SHA => PSK-AES128-CBC-SHA
[DEBUG] 2018-01-31 17:05:06.787 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_PSK_WITH_AES_128_CBC_SHA => PSK-AES128-CBC-SHA
[DEBUG] 2018-01-31 17:05:06.787 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_RSA_WITH_AES_256_CBC_SHA => AES256-SHA
[DEBUG] 2018-01-31 17:05:06.787 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_RSA_WITH_AES_256_CBC_SHA => AES256-SHA
[DEBUG] 2018-01-31 17:05:06.787 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_RSA_WITH_AES_256_CBC_SHA256 => AES256-SHA256
[DEBUG] 2018-01-31 17:05:06.788 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_RSA_WITH_AES_256_CBC_SHA256 => AES256-SHA256
[DEBUG] 2018-01-31 17:05:06.788 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_PSK_WITH_AES_256_CBC_SHA => PSK-AES256-CBC-SHA
[DEBUG] 2018-01-31 17:05:06.788 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_PSK_WITH_AES_256_CBC_SHA => PSK-AES256-CBC-SHA
[DEBUG] 2018-01-31 17:05:06.788 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: TLS_RSA_WITH_3DES_EDE_CBC_SHA => DES-CBC3-SHA
[DEBUG] 2018-01-31 17:05:06.788 [[main]-pipeline-manager] CipherSuiteConverter - Cipher suite mapping: SSL_RSA_WITH_3DES_EDE_CBC_SHA => DES-CBC3-SHA
[DEBUG] 2018-01-31 17:05:06.789 [[main]-pipeline-manager] OpenSsl - Supported protocols (OpenSSL): [[SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2]]
[DEBUG] 2018-01-31 17:05:06.789 [[main]-pipeline-manager] OpenSsl - Default cipher suites (OpenSSL): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA]
[DEBUG] 2018-01-31 17:05:06.806 [[main]-pipeline-manager] MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 4
[DEBUG] 2018-01-31 17:05:06.861 [[main]-pipeline-manager] NioEventLoop - -Dio.netty.noKeySetOptimization: false
[DEBUG] 2018-01-31 17:05:06.861 [[main]-pipeline-manager] NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
[DEBUG] 2018-01-31 17:05:06.875 [[main]-pipeline-manager] PlatformDependent - org.jctools-core.MpscChunkedArrayQueue: available
[DEBUG] 2018-01-31 17:05:06.892 [[main]-pipeline-manager] SslSimpleBuilder - Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
[DEBUG] 2018-01-31 17:05:06.894 [[main]-pipeline-manager] SslSimpleBuilder - Cipher is supported: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
[DEBUG] 2018-01-31 17:05:06.894 [[main]-pipeline-manager] SslSimpleBuilder - Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
[DEBUG] 2018-01-31 17:05:06.894 [[main]-pipeline-manager] SslSimpleBuilder - Cipher is supported: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[DEBUG] 2018-01-31 17:05:06.894 [[main]-pipeline-manager] SslSimpleBuilder - Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
[DEBUG] 2018-01-31 17:05:06.894 [[main]-pipeline-manager] SslSimpleBuilder - Cipher is supported: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
[DEBUG] 2018-01-31 17:05:06.894 [[main]-pipeline-manager] SslSimpleBuilder - Cipher is supported: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[DEBUG] 2018-01-31 17:05:06.894 [[main]-pipeline-manager] SslSimpleBuilder - Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
[INFO ] 2018-01-31 17:05:06.944 [[main]<beats] Server - Starting server on port: 5044

(Pier-Hugues Pellerin) #8

@Luke_Briner This look like a bug to me, can you create an issue on https://github.com/logstash-plugins/logstash-input-beats?


(Luke Briner) #9

I have raised a ticket. The debugging is confusing because curl seems to complain it couldn't agree a handshake even though Logstash logs that it had chosen the correct cipher to handshake with! Let's hope someone with better debugging tools can reproduce it!
Thanks


(system) #10

This topic was automatically closed after 21 days. New replies are no longer allowed.