Filebeat is still sending logs to logstash, even curl ssl is failing

Elastic Stack Beats
1

Hi,
I ran into a strange problem with filebeat. I have filebeat monitoring logs in mulitple instances shipping them to central logstash. After few days, some of those filebeat dropped connection to logstash and i couldnt see logs from them in kibana. I went to those boxes and saw the folloiwing in thier log files.
2016-07-29T00:39:16-07:00 ERR SSL client failed to connect with: read tcp localip:37799->centrallogstaship:443: i/o timeout
2016-07-29T00:40:42-07:00 ERR SSL client failed to connect with: read tcp localip:44914->centrallogstaship:443: read: connection reset by peer
2016-07-29T00:42:12-07:00 ERR SSL client failed to connect with: dial tcp centrallogstaship:443: i/o timeout
I understood its an ssl issue and inorder to check I tried curl -v --cacert logstash-farwarder.crt https://centrallogstaship:443 And ssl handshake failed.
This explains the issue above. But problem is, initially with same certificate from same box i could send logs to logstash with filebeat. And with the same certificate to the same logstash other filebeats are still sending logs without fail. And more strange thing is just to check ssl from the working box. I tried to run curl command, and ssl handshake is failing, but still filebeat is sending logs. How could this be possible.

2

is certificate valid or expired? Does host in certificate match remote?

3

Yes, I checked with the following command
openssl x509 -text -noout -in logstash-farwarder.crt
and the validity is till 2026 and subject aleternate name matches with the logstash's ip.
I followed this method to create certificate for logstash
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
But this certificate is causing the failure even though its valid till 2026.
Can someone help me on how to actually setup logstash and filebeat connection over proper ssl certificates, which wont fail.

4

have you seen this docs: https://www.elastic.co/guide/en/beats/filebeat/current/configuring-tls-logstash.html

Whats the exact output of curl when trying to connect. If certificate is correct, how come curl still fails the handshake?

5

Yes i went through that doc, But in order to implement that i need a CA. Instead i followed this: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04 and used self signed certificate to make the connection. Initially it worked well, But now i am seeing these ssl issues. When i try ssl handshake with curl -v --cacert 52.35.110.46.crt https://52.35.110.46:443

  • About to connect() to 52.35.110.46 port 443 (#0)
  • Trying 52.35.110.46...
  • connected
  • Connected to 52.35.110.46 (52.35.110.46) port 443 (#0)
  • successfully set certificate verify locations:
  • CAfile: 52.35.110.46.crt
    CApath: none
  • SSLv3, TLS handshake, Client hello (1):
  • Unknown SSL protocol error in connection to 52.35.110.46:443
  • Closing connection #0
    curl: (35) Unknown SSL protocol error in connection to 52.35.110.46:443
    Its ending like this.

This is the output even on boxes where filebeat is still monitoring events and pushing to logstash.

6 
* Unknown SSL protocol error in connection to 52.35.110.46:443 * Closing connection #0

Unknown SSL protocol? Can you share your filebeat output config and your logstash input config? Did you compare your different filebeat configs?

7

Following is my filbeat output config

output:
    logstash:
      hosts: ["52.35.110.46:443"]
      loadbalance: true
      tls:
         certificate_authorities: [/sc/filebeat/logstash-forwarder.crt]

Following is my logstash config :

input {
        beats {
                port => 443
                ssl => true
                ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
                ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
        }
}
8

hm weird. All filebeat instances using the same output configuration?

Thing is, curl should not fail with protocol error. Anything in logstash logs?

Can you try the curl command with different TLS versions? From curl man-page:

You can use options --tlsv1.0, --tlsv1.1, and --tlsv1.2 to control the TLS version more precisely (if the SSL backend in use supports such a level of control).

9

All filebeat instances using the same config.
logstash logs are empty.
i went through curl man page and found only ssl2, ssl3 and tls1 options and with all those options, i am ending up with same error.
I made a test in my local instance.
I started logstash with old certificate and used curl to connect and failed.
Then i created a new certificate following the same process, started logstash with new certificate and tried connecting with curl. And curl succesfully made ssl handshake, just to test i used ssl2, ssl3 and tls1 options with curl to connect and it failed in first two cases and established handshake with tls1.
From this i am deducing, on jun3 when i first created old certificate, openssl is using some ssl3 version, and when i copied that certificate and tried curl, it worked. But later ssl version got upgraded somehow to tls1 and as there is no data for some period of time, so filebeat dropped connection to logstash and when new data came, filebeat tried establishing connection but couldn't, as ssl version got changed, and thats the reason why i couldnt curl with that certificate currently. And when i create a new certificate now. It created with latest tls1 version and now with that certificate, filebeat could establish connection to logstash.
However, this is a complete guess, as i dont know how much this could be true and how I can prove this theory.

10

It could be somehow related, as filebeat asks for TLS at least being 1.1 by default (for security concerns on SSLv3) I think.

11

Yeah that could be a reason, probabaly my aws box would have been 1.1 and now it got updated to 1.2, and made the certificate invalid causing the whole issue. Fine, i will go ahead and have a proper ca and use certificates of tls1.2 signed by it for two way authentication. Probably it will solve the issue.
But one thing i want to confirm from filebeat side is filebeat will keep on monitoring the files, and if there is no data for some specified amount of time, it will drop the connection. And when the new data appeared, it will retry the connection. Am I correct?.

12

As long as filebeat is running, it will try to send new data.

13

  1. TLS versions can be quite a pain. For TLS 1.2 You need at least JVM 8 I think (that's why beats asks for minimum 1.1 by default, to be compatible to older JVMs). The versions support can also be configured in beats logstash output section.

  2. Yes, whenever beats finds it can not send, it will attempt to reconnect. Plus filebeat is designed to never drop any log lines (as long as it still has access to files). Logstash can be restarted for maintenance reasons and when up again filebeat will just continue sending events.

14

Thanks steffens.

15

This topic was automatically closed after 21 days. New replies are no longer allowed.