Logstash Certificate Error "Looks like you either have an invalid key..."

[bfrd (bradford)]

Hello there,

I am installing the ELK stack on a new server, everything seems to be going well except for logstash. From the ELK server's /var/log/logstash/logstash-plain.log am getting:
[ERROR][logstash.inputs.beats ] Looks like you either have an invalid key or your private key was not in PKCS8 format. {:exception=>java.lang.IllegalArgumentException: File does not contain valid private key: /etc/pki/tls/private/logstash-forwarder.key}

From remote clients /var/log/syslog running filebeat i'm getting:
/usr/bin/filebeat[6968]: transport.go:125: SSL client failed to connect with: read tcp <filebeat-client>:37342-><elk-server>:5044: i/o timeout and I get essentially the same message from winlogbeat:
ERR Connecting error publishing events (retrying): read tcp <client-ip>:61676-><server-ip>:5044: i/o timeout

server's /etc/logstash/conf.d/02-beats-input.conf (assume indentation):
input { beats { port => 5044 ssl => true ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } }

I generated the keypair with:
openssl req -subj '/CN=<servers-fqdn>/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

I have tried creating the keypair with the CN (Common Name (hostname)) and the FQDN as suggested above.

Is there actually something wrong with the SSL portion of my setup or something else? I am on Debian 9 Stretch amd64, using package default-jre (openjdk-8-jre).

May be unrelated but if I do systemctl status logstash -l I see:
Jul 07 11:49:21 <server> systemd[1]: Started logstash. Jul 07 11:49:35 <server> logstash[9609]: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Jul 07 11:49:37 <server> logstash[9609]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties Jul 07 11:49:38 <server> logstash[9609]: log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAuthCache). Jul 07 11:49:38 <server> logstash[9609]: log4j:WARN Please initialize the log4j system properly. Jul 07 11:49:38 <server> logstash[9609]: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Despite the messages above the service is running, stays running, and netstat -plant | grep 5044 shows java listening on that port... only for ipv6 though it seems.
tcp6 0 0 :::5044 :::* LISTEN 9609/java

Elasticsearch, Kibana, and Logstash were installed using the .deb packages provided by elastic.co today...
elasticsearch-5.5.0.deb
kibana-5.5.0-amd64.deb
logstash-5.5.0.deb

Thanks for reading and thanks in advance for the help!
-Bradford

[bfrd (bradford)]

I read in Debian 9 changelogs that OpenSSL has been updated. Regenerated the keypair using OpenSSL from the old Debian 8 server, imported them to Debian 9 into their respective directories /etc/pki/tls/{certs, private}, restarted the services, distributed the certificates to my clients running filebeat and winlogbeat and now everything is flowing.

So this issue is resolved by using an older version of OpenSSL to generate the public/private keypair that is used to encrypt the data between beat clients and logstash.

There may be a set of switches/options that you can pass to the newer OpenSSL that will generate a compatible keypair, if anyone knows what those are you can post them here for people that might have this issue in the future.

---

Command:
openssl req -subj '/CN=<fqdn>/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

Works from:
OpenSSL from Debian 8 ( "Jessie" ):
openssl 1.0.1t-1+deb8u6

Does Not Work from:
OpenSSL from Debian 9 ( "Stretch" ):
openssl 1.1.0f-3

[warkolm (Mark Walkom)]

Thanks for sharing your solution with us!

[theuntergeek (Aaron Mildenstein)]

Indeed, thanks for sharing. The OpenSSL library change in Debian 9 is likely to bite a lot of users.

Curator's SSL will work if you install via pip, but not the current (as of today) DEB packages, due to them being based on the OpenSSL 1.0 library. I'm going to have to make a special release based on the 1.1 library.