# Filebeat is still sending logs to logstash, even curl ssl is failing

**URL:** <https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 29, 2016, 8:54am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709 "2016-07-29T08:54:59Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\
**Post date:** [July 29, 2016, 8:54am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/1 "2016-07-29T08:54:59Z")

</div>

Hi,  
I ran into a strange problem with filebeat. I have filebeat monitoring logs in mulitple instances shipping them to central logstash. After few days, some of those filebeat dropped connection to logstash and i couldnt see logs from them in kibana. I went to those boxes and saw the folloiwing in thier log files.  
2016-07-29T00:39:16-07:00 ERR SSL client failed to connect with: read tcp localip:37799-\>centrallogstaship:443: i/o timeout  
2016-07-29T00:40:42-07:00 ERR SSL client failed to connect with: read tcp localip:44914-\>centrallogstaship:443: read: connection reset by peer  
2016-07-29T00:42:12-07:00 ERR SSL client failed to connect with: dial tcp centrallogstaship:443: i/o timeout  
I understood its an ssl issue and inorder to check I tried curl -v --cacert logstash-farwarder.crt [https://centrallogstaship:443](https://centrallogstaship:443) And ssl handshake failed.  
This explains the issue above. But problem is, initially with same certificate from same box i could send logs to logstash with filebeat. And with the same certificate to the same logstash other filebeats are still sending logs without fail. And more strange thing is just to check ssl from the working box. I tried to run curl command, and ssl handshake is failing, but still filebeat is sending logs. How could this be possible.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [July 29, 2016, 12:12pm UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/2 "2016-07-29T12:12:44Z")

</div>

is certificate valid or expired? Does host in certificate match remote?

---

<div class="post-metadata">

**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\
**Post date:** [August 1, 2016, 4:43am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/3 "2016-08-01T04:43:34Z")

</div>

Yes, I checked with the following command  
openssl x509 -text -noout -in logstash-farwarder.crt  
and the validity is till 2026 and subject aleternate name matches with the logstash's ip.  
I followed this method to create certificate for logstash  
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt  
But this certificate is causing the failure even though its valid till 2026.  
Can someone help me on how to actually setup logstash and filebeat connection over proper ssl certificates, which wont fail.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 2, 2016, 10:33am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/4 "2016-08-02T10:33:51Z")

</div>

have you seen this docs: [https://www.elastic.co/guide/en/beats/filebeat/current/configuring-tls-logstash.html](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-tls-logstash.html)

Whats the exact output of curl when trying to connect. If certificate is correct, how come curl still fails the handshake?

---

<div class="post-metadata">

**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\
**Post date:** [August 2, 2016, 11:53am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/5 "2016-08-02T11:53:21Z")

</div>

Yes i went through that doc, But in order to implement that i need a CA. Instead i followed this: [https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04) and used self signed certificate to make the connection. Initially it worked well, But now i am seeing these ssl issues. When i try ssl handshake with curl -v --cacert 52.35.110.46.crt [https://52.35.110.46:443](https://52.35.110.46:443)

- About to connect() to 52.35.110.46 port 443 (#0)
- Trying 52.35.110.46...
- connected
- Connected to 52.35.110.46 (52.35.110.46) port 443 (#0)
- successfully set certificate verify locations:
- CAfile: 52.35.110.46.crt  
CApath: none
- SSLv3, TLS handshake, Client hello (1):
- Unknown SSL protocol error in connection to 52.35.110.46:443
- Closing connection #0  
curl: (35) Unknown SSL protocol error in connection to 52.35.110.46:443  
Its ending like this.

This is the output even on boxes where filebeat is still monitoring events and pushing to logstash.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 2, 2016, 12:25pm UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/6 "2016-08-02T12:25:18Z")

</div>

```auto
* Unknown SSL protocol error in connection to 52.35.110.46:443 * Closing connection #0

```

Unknown SSL protocol? Can you share your filebeat output config and your logstash input config? Did you compare your different filebeat configs?

---

<div class="post-metadata">

**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\
**Post date:** [August 3, 2016, 4:50am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/7 "2016-08-03T04:50:24Z")

</div>

Following is my filbeat output config

```auto
output:
    logstash:
      hosts: ["52.35.110.46:443"]
      loadbalance: true
      tls:
         certificate_authorities: [/sc/filebeat/logstash-forwarder.crt]

```

Following is my logstash config :

```auto
input {
        beats {
                port => 443
                ssl => true
                ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
                ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
        }
}

```

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 3, 2016, 11:24am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/8 "2016-08-03T11:24:09Z")

</div>

hm weird. All filebeat instances using the same output configuration?

Thing is, curl should not fail with protocol error. Anything in logstash logs?

Can you try the curl command with different TLS versions? From curl man-page:

> You can use options --tlsv1.0, --tlsv1.1, and --tlsv1.2 to control the TLS version more precisely (if the SSL backend in use supports such a level of control).

---

<div class="post-metadata">

**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\
**Post date:** [August 3, 2016, 11:51am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/9 "2016-08-03T11:51:25Z")

</div>

All filebeat instances using the same config.  
logstash logs are empty.  
i went through curl man page and found only ssl2, ssl3 and tls1 options and with all those options, i am ending up with same error.  
I made a test in my local instance.  
I started logstash with old certificate and used curl to connect and failed.  
Then i created a new certificate following the same process, started logstash with new certificate and tried connecting with curl. And curl succesfully made ssl handshake, just to test i used ssl2, ssl3 and tls1 options with curl to connect and it failed in first two cases and established handshake with tls1.  
From this i am deducing, on jun3 when i first created old certificate, openssl is using some ssl3 version, and when i copied that certificate and tried curl, it worked. But later ssl version got upgraded somehow to tls1 and as there is no data for some period of time, so filebeat dropped connection to logstash and when new data came, filebeat tried establishing connection but couldn't, as ssl version got changed, and thats the reason why i couldnt curl with that certificate currently. And when i create a new certificate now. It created with latest tls1 version and now with that certificate, filebeat could establish connection to logstash.  
However, this is a complete guess, as i dont know how much this could be true and how I can prove this theory.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 3, 2016, 12:07pm UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/10 "2016-08-03T12:07:09Z")

</div>

It could be somehow related, as filebeat asks for TLS at least being 1.1 by default (for security concerns on SSLv3) I think.

---

<div class="post-metadata">

**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\
**Post date:** [August 4, 2016, 4:40am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/11 "2016-08-04T04:40:46Z")

</div>

Yeah that could be a reason, probabaly my aws box would have been 1.1 and now it got updated to 1.2, and made the certificate invalid causing the whole issue. Fine, i will go ahead and have a proper ca and use certificates of tls1.2 signed by it for two way authentication. Probably it will solve the issue.  
But one thing i want to confirm from filebeat side is filebeat will keep on monitoring the files, and if there is no data for some specified amount of time, it will drop the connection. And when the new data appeared, it will retry the connection. Am I correct?.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [August 8, 2016, 7:44am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/12 "2016-08-08T07:44:29Z")

</div>

As long as filebeat is running, it will try to send new data.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [August 8, 2016, 12:22pm UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/13 "2016-08-08T12:22:27Z")

</div>

1. TLS versions can be quite a pain. For TLS 1.2 You need at least JVM 8 I think (that's why beats asks for minimum 1.1 by default, to be compatible to older JVMs). The versions support can also be configured in beats logstash output section.

2. Yes, whenever beats finds it can not send, it will attempt to reconnect. Plus filebeat is designed to never drop any log lines (as long as it still has access to files). Logstash can be restarted for maintenance reasons and when up again filebeat will just continue sending events.

---

<div class="post-metadata">

**Author:** ![Sai\_Birada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sai_birada/32/142367_2.png) [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\
**Post date:** [August 9, 2016, 3:28am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/14 "2016-08-09T03:28:55Z")

</div>

Thanks steffens.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 19, 2016, 8:55am UTC](https://discuss.elastic.co/t/filebeat-is-still-sending-logs-to-logstash-even-curl-ssl-is-failing/56709/15 "2016-08-19T08:55:11Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.