Logstash SSL File does not contain a valid private key with Beats

[kharvey (Ken Harvey)]

I am trying to setup filebeat to use SSL, but I keep getting errors on the logstash server.

I keep receiving an error when I have my input on logstash set to Beats.

java.lang.IllegalArgumentException: File does not contain valid private key: /etc/logstash/ssl/logstash-proxy.key

Here is my input on my logstash:

input {
  beats {
    #tcp {
    port => 5044

    #ssl_enable => true
    #ssl_verify => false
    #ssl_key => "/etc/logstash/ssl/logstash-proxy.key"
    #ssl_cert => "/etc/logstash/ssl/logstash-proxy.crt"
    #ssl_extra_chain_certs => "/etc/logstash/ssl/ca.com.crt"

    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/ssl/ca.com.crt"]
    ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt"
    ssl_key => "/etc/logstash/ssl/logstash-proxy.key"
    ssl_verify_mode => "none"
  }
}

If I run this command while logstash input is set to Beats

curl -k -v --cacert logstash-proxy.crt https://10.1.2.143:5044

I receive this error on the filebeat client:

* About to connect() to 10.1.2.143 port 5044 (#0)
*   Trying 10.1.2.143...
* Connected to 10.1.2.143 (10.1.2.143) port 5044 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* NSS error -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
* Closing connection 0
curl: (35) TCP connection reset by peer

But if I switch over to a TCP configuration, and I run the same curl I receive this on the filebeat client

* About to connect() to 10.1.2.143 port 5044 (#0)
*   Trying 10.1.2.143...
* Connected to 10.1.2.143 (10.1.2.143) port 5044 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.client.com,E=devops@XYZ.com,OU=ZYZ DevOps Team,O=XYZ,L=City,ST=NY,C=US
*       start date: Dec 05 18:44:03 2018 GMT
*       expire date: Dec 04 18:44:03 2023 GMT
*       common name: *.client.com
*       issuer: CN=ca.com,E=devops@XYZ.com,OU=XYZ DevOps Team,O=XYZ,L=City,ST=NY,C=US
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.1.2.143:5044
> Accept: */*

If I leave my input set at Beats and I send something from my beats client I receive this error on the logstash console

[2019-03-20T14:24:04,106][WARN ][org.logstash.beats.Server] Exception caught in channel initializer
java.lang.IllegalArgumentException: File does not contain valid private key: /etc/logstash/ssl/logstash-proxy.key

But if I switch to TCP on my logstash input and I try to send something to it using beats I receive this error:

[2019-03-20T14:22:43,832][ERROR][logstash.inputs.tcp      ] Error in Netty pipeline: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE

I have these certificates working for NXLog -> logstash -> elasticsearch.
Right now I am attempting to replace NXLog with filebeats, and the final piece, which I thought would be the easiest, was to enable SSL. And I can't get it to work.

Any suggestions?

[kharvey (Ken Harvey)]

I had to convert the .key to pkcs8 by running:

openssl pkcs8 -in logstash-proxy.key -topk8 -out logstash-proxy-pkcs8.key -nocrypt

I then changed my logstash input to be

input {
  beats {
    port => 5044

    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/ssl/ca.com.crt"]
    ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt"
    ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key"
    ssl_verify_mode => "none"
  }
}

When I made this change I had an error of

[2019-03-20T15:55:53,695][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 10.1.2.122:53694] Handling exception: Connection reset by peer

When I ran filebeat in verbose I received this error:

2019-03-20T15:45:40.452-0700    ERROR   pipeline/output.go:100  Failed to connect to backoff(async(tcp://10.1.2.143:5044)): x509: cannot validate certificate for 10.1.2.143 because it doesn't contain any IP SANs

This was a quick fix. I added my test server to the hosts file on the local machine and used the "DNS" name in the output on my filebeat.

Everything is now working. Yay!