Filebeat to Logstash TLS handshake failure

Elastic Stack Beats
1

EDIT: I'm running version 5.3 of everything.

I'm currently trying to get my filebeat to logstash connections using SSL, but I keep running into the error:

2017/04/05 16:42:16.045679 sync.go:53: DBG  connect
2017/04/05 16:42:16.108391 single.go:140: ERR Connecting error publishing events (retrying): remote error: tls: handshake failure
2017/04/05 16:42:16.108424 single.go:156: DBG  send fail

I can ship logs without SSL enabled fine. And I've confirmed that the connections are reaching the Logstash machine with SSL configured, just not actually able to establish a functioning connection. I can also create an openssl client connection from the beats machine to the logstash machine on port 5044 and the SSL certs and connection all checkout fine.

I have Logstash, Elasticsearch and Kibana all communicating via ssl connections just fine, but Filebeats just won't cooperate. My current filebeats config is:

name: "shipper"

filebeat:
  prospectors:
    - input_type: log
      tags: [ "mylogs" ]
      tail_files: true
      paths:
        - /path/to/logs
        - /path/to/logs
        - /path/to/logs

output:
  logstash:
    hosts: [ "logstash.host:5044" ]
    compression_level: 1
    ssl:
      certificate_authorities: [ "/path/to/ca.crt" ]

logging:
  level: warning
  to_files: true
  to_syslog: false
  files:
    path: /path/to/logs
    name: filebeats.log
    keepfiles: 7

My logstash input on the other end is:

input {
  beats {
    host => "xxx.xxx.xxx.xxx"
    port => 5044
    ssl => true
    ssl_certificate_authorities => [ "/path/to/ca.crt" ]
    ssl_certificate => "/path/to/logstash.crt"
    ssl_key => "/path/to/logstash.pkcs8"
  }
}
2

Are you sure the host setting in the Beats input in Logstash is what you want? Try removing that, perhaps it's not binding to the correct interface with the value you provided.

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-host

3

No change in behavior if I remove the host setting.

4

Why is it you have configured ssl_certificate_authorities in logstash? This is only required if client authentication is to be enabled.

Have you tried to validate your certificate e.g. with curl or openssl client? (See FAQ)

5

ssl_certficate_authorities is set because I was following the config instructions provided by elastic, and trying anything and everything to make the logstash -> beat connection function.

I have tested the ssl cert via openssl as stated in my original post:

This cert is also used to connect with elasticsearch and is working.

6

Example of working SSL connection via the FAQ troubleshooting example:

curl -v --cacert ca-bundle.crt https://<redacted>:5044
* About to connect() to <redacted> port 5044 (#0)
*   Trying <redacted>... connected
* Connected to <redacted> (<redacted>) port 5044 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*  subject: E=<redacted>
*  start date: Mar 18 22:45:04 2015 GMT
*  expire date: Mar 19 22:45:04 2018 GMT
*  common name: <redacted>
*  issuer: <redacted>
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: <redacted>:5044
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host <redacted> left intact
curl: (52) Empty reply from server
* Closing connection #0
7

This is a shot in the dark, but does ca-bundle.crt contain multiple certs? Have you tried splitting them into individual files and providing each file in the certificate_authorities list in Filebeat?

Also, does Filebeat work if you disable certificate verification using verification_mode: none?

1 Like
8

YES! It was the bundled CA causing the issue. Split them out and added to the authorities setting and everything is joy.

Thank you so very much!

9

Hm, that's funny. Which format is your ca-bundle.crt? If I remember correctly, the PEM-reader in stdlib will iterate and add all certificates to the CA-Certificate-Set (being a set, even order should not matter).

10

X.509v3 PEM format. It's the CA bundle that we use for everything that validates SSL, which is why it didn't even occur to me it might be the issue. Never had a problem before.

11

hm, that's weird. I'd treat this as a bug. Can you open an issue with github including a fake-certificate bundle for testing ? Or shell commands to produce a fake certificate bundle matching your bundle in structure.

12

This topic was automatically closed after 21 days. New replies are no longer allowed.