# Filebeat could not connect to logstash on SSL

**URL:** <https://discuss.elastic.co/t/filebeat-could-not-connect-to-logstash-on-ssl/152723>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 16, 2018, 9:27pm UTC](https://discuss.elastic.co/t/filebeat-could-not-connect-to-logstash-on-ssl/152723 "2018-10-16T21:27:37Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [October 16, 2018, 9:27pm UTC](https://discuss.elastic.co/t/filebeat-could-not-connect-to-logstash-on-ssl/152723/1 "2018-10-16T21:27:37Z")

</div>

Hello there,

I set up the SSL on all our ELK nodes (2 Elastinodes, L&K on another node), all work fine.  
Without setting up SSL on Beat hosts, everything worked fine.  
Now we were requested to set up the SSL/TLS between the Beat servers and Logstash, but ran into issues.

Here are the configuration on filebeat.yml for SSL:

```auto
output.logstash:
  # The Logstash hosts
  hosts: ["logstashhostname:5044"]
  ssl.certificate_authorities: ["/opt/keys/ChainBundle2.crt"]
  ssl.certificate: "/opt/keys/ServerCertificate.crt"
  ssl.key: "/opt/keys/xxxx-key.pem"

here is the pipeline configuration file for Beats:

input {
  beats {
    port => 5044
    client_inactivity_timeout => 120
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/keys/ChainBundle2.crt"]
    ssl_certificate => "/etc/logstash/keys/ServerCertificate.crt"
    ssl_key => "/etc/logstash/keys/xxxx-key.pem"
    ssl_verify_mode => "force_peer"
 }
}

# The filter part of this file is commented out to indicate that it
# is optional.
# filter {
#
# }

output {
  elasticsearch {
    user => "logstash_ingest"
    password => "changeme"
    ssl => true
    ssl_certificate_verification => true
    cacert => "/etc/logstash/keys/ChainBundle2.crt"
    action => "index"
    hosts => ["elasticnodehostname"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

```

====================

Here is the errors in logstash.log (there is only one filebeat connecting to logstash now for testing\_:

```auto
[2018-10-16T21:16:16,593][ERROR][logstash.inputs.beats] Looks like you either have a bad certificate, an invalid key or your private key was not in PKCS8 format.
[2018-10-16T21:16:16,593][WARN][io.netty.channel.ChannelInitializer] Failed to initialize a channel. Closing: [id: 0x396d22d0, L:/10.100.35.68:5044 - R:/10.100.12.118:41314]
java.lang.IllegalArgumentException: File does not contain valid private key: /etc/logstash/keys/hls-201710-dxc-key.pem
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
........
Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1045) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
       ... 20 more
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
        at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169) ~[sunec.jar:1.8.0_181]
        at java.security.KeyFactory.generatePrivate(KeyFactory.java:372) ~[?:1.8.0_181]
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1043) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        ... 20 more
Caused by: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
        at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:352) ~[?:1.8.0_181]
        at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:357) ~[?:1.8.0_181]
        at sun.security.ec.ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:73) ~[sunec.jar:1.8.0_181]
........

```

===========================

Here is the errors on filebeat.log on filebeat server where we have hadoop running on:

```auto
2018-10-16T15:46:14.647-0500 INFO log/harvester.go:251 Harvester started for file: /var/log/messages
2018-10-16T15:46:14.696-0500 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://Logstashhostname:5044))
2018-10-16T15:46:15.721-0500 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://Logstashhostname:5044)): read tcp filebeatserverIP:58254->logstashserverIP:5044: read: connection reset by peer
2018-10-16T15:46:15.721-0500 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://Logstashhostname:5044)) with 1 reconnect attempt(s)
2018-10-16T15:46:17.732-0500 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://Logstashhostname:5044)): read tcp filebeatserverIP:58264->logstashserverIP:5044
....

```

======================================

Any thing that is missing or incorrect? Please help.

Thanks in advance

Li

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [October 17, 2018, 11:48am UTC](https://discuss.elastic.co/t/filebeat-could-not-connect-to-logstash-on-ssl/152723/2 "2018-10-17T11:48:28Z")

</div>

See the logstash logs. Logstash didn't start correctly. Filebeat is complaining about Logstash not being available.

Logstash requires you to provide the private key in PKCS8 format.

---

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [October 18, 2018, 12:26am UTC](https://discuss.elastic.co/t/filebeat-could-not-connect-to-logstash-on-ssl/152723/3 "2018-10-18T00:26:24Z")

</div>

Thanks for your input:

here is what I did:

1: openssl pkcs8 -nocrypt -topk8 -inform PEM -outform PEM -in hls-201710-dxc-key.pem -out hls-201710-dxc-pkcs8.pem

1. use this pkcs8 format key in beat pipeline.conf as below:

input {  
beats {  
port =\> 5044  
client\_inactivity\_timeout =\> 120  
ssl =\> true  
ssl\_certificate\_authorities =\> ["/opt/keys/ChainBundle2.crt"]  
ssl\_certificate =\> "/opt/keys/ServerCertificate.crt"  
ssl\_key =\> "/opt/keys/xxx-201710-xxx-pkcs8.pem"  
ssl\_verify\_mode =\> "force\_peer"  
}  
}

And use the pcks format key: xxx-201710-xxx-pkcs8.pem still the same error as below… any thought?

[2018-10-17T18:35:37,408][ERROR][logstash.inputs.beats] Looks like you either have a bad certificate, an invalid key or your private key was not in PKCS8 format.

[2018-10-17T18:35:37,504][ERROR][logstash.inputs.beats] Looks like you either have a bad certificate, an invalid key or your private key was not in PKCS8 format.

[2018-10-17T18:35:37,520][ERROR][logstash.inputs.beats] Looks like you either have a bad certificate, an invalid key or your private key was not in PKCS8 format.

[2018-10-17T18:35:37,574][ERROR][logstash.inputs.beats] Looks like you either have a bad certificate, an invalid key or your private key was not in PKCS8 format.

[2018-10-17T18:35:37,751][ERROR][logstash.inputs.beats] Looks like you either have a bad certificate, an invalid key or your private key was not in PKCS8 format.

Anything I didn't do correctly?

This is our showstopper now...

Thanks a lot

Li

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [October 19, 2018, 12:24pm UTC](https://discuss.elastic.co/t/filebeat-could-not-connect-to-logstash-on-ssl/152723/4 "2018-10-19T12:24:24Z")

</div>

Unfortunately the Logstash error message is not very helpful.

Have you tried to test your certificates + keys with `openssl s_server` (create server endpoint) and `openssl s_client` (client used to debug connection and TLS setup in conjunction with s\_server)?

See:

- [https://www.openssl.org/docs/man1.0.2/apps/openssl-s\_server.html](https://www.openssl.org/docs/man1.0.2/apps/openssl-s_server.html)
- [https://www.openssl.org/docs/man1.0.2/apps/openssl-s\_client.html](https://www.openssl.org/docs/man1.0.2/apps/openssl-s_client.html)

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [October 19, 2018, 12:43pm UTC](https://discuss.elastic.co/t/filebeat-could-not-connect-to-logstash-on-ssl/152723/5 "2018-10-19T12:43:45Z")

</div>

Once you converted the private key to pkcs8, I see you're still seeing the "Looks like you either have a bad certificate.." log message, are you also still seeing java stack traces that you could post? those provide extra insight into what is causing the issue.

Also, I see that you have setup ssl\_key, ssl\_certificate and ssl\_certificate\_authorities the same way in beats input and filebeat, is this intended? can you explain how your pki is setup?

---

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [November 2, 2018, 2:12pm UTC](https://discuss.elastic.co/t/filebeat-could-not-connect-to-logstash-on-ssl/152723/6 "2018-11-02T14:12:34Z")

</div>

Thanks, this has been resolved... this was due to the certificates and keys I used were not right.  
Our Security team helped to fixed this.

Thanks again for your help

Li

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 30, 2018, 4:12pm UTC](https://discuss.elastic.co/t/filebeat-could-not-connect-to-logstash-on-ssl/152723/7 "2018-11-30T16:12:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.