Filebeat cannot connect to Logstash using ssl. curl: (35) TCP connection reset by peer

Elastic Stack Beats
1

Hi guys,

i've set up security for ELK. Everythings works, except the communication between Filebeat and Logstash. Everything is version 7.4

filebeat.yml

output.logstash:
  # The Logstash hosts
  hosts: ["ip-address:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/certs/ca.crt"]

  # Certificate for SSL client authentication
  ssl.certificate: "/certs/filebeat_client.crt"

  # Client Certificate Key
  ssl.key: "/filebeat_client.key"
  ssl.key_passphrase: password

logstash config file:

input {
  beats {
    port => "5044"
	ssl => true
	ssl_certificate_authorities => ["/certs/logstash-01/ca.crt"]
    ssl_certificate => "/certs/logstash-01/logstash-01.crt"
    ssl_key => "/certs/logstash-01/logstash01.pkcs8.key"
	ssl_verify_mode => "peer"
	
  }
}

filter {
  csv {
    separator => ","
    columns => ["id","insert_time","unix_time"]
	convert => {
	  "id" => "integer"
	}
  }
}

output {
  elasticsearch {
    hosts => ["elastic01.com:9200"]
	index => "%{[@metadata][beat]}-%{[@metadata][version]}"
	ssl => true
	ssl_certificate_verification => true
	cacert => '/certs/logstash-01/elastic-ca.pem'
	user => logstash_internal
    password => password
  }
}

When I try to start filebeat, I see following error in the filebeat logs:

|2019-12-17T16:01:20.284Z|INFO|pipeline/output.go:95|Connecting to backoff(async(tcp://logstash-ip-address:5044))|
|---|---|---|---|
|2019-12-17T16:01:21.434Z|ERROR|pipeline/output.go:100|Failed to connect to backoff(async(tcp://logstash-ip-address:5044)): read tcp filebeat-machine-ip-addres:54108->logstash-ip-address:5044: read: connection reset by peer|

following in the logstash logs:

[2019-12-17T16:01:20,304][INFO ][org.logstash.beats.BeatsHandler][main] [local: 0.0.0.0:5044, remote: undefined] Handling exception: javax.net.ssl.SSLException: java.lang.Exception: Error setting private key (error:0b000074:X.509 certificate routines:OPENSSL_internal:KEY_VALUES_MISMATCH)
[2019-12-17T16:01:20,304][INFO ][org.logstash.beats.BeatsHandler][main] [local: 0.0.0.0:5044, remote: undefined] Handling exception: javax.net.ssl.SSLException: java.lang.Exception: Error setting private key (error:0b000074:X.509 certificate routines:OPENSSL_internal:KEY_VALUES_MISMATCH)
[2019-12-17T16:01:20,305][WARN ][io.netty.channel.DefaultChannelPipeline][main] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: java.lang.Exception: Error setting private key (error:0b000074:X.509 certificate routines:OPENSSL_internal:KEY_VALUES_MISMATCH)

I tried to:

curl -v --cacert ./certs/ca.crt https://logstash-ip-address:5044

and I received following:

* About to connect() to logstash-ip-address port 5044 (#0)
*   Trying logstash-ip-address...
* Connected to logstash-ip-address (logstash-ip-address) port 5044 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /certs/ca.crt
  CApath: none
* NSS error -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
* Closing connection 0
curl: (35) TCP connection reset by peer

Info:

I've extracted the ca.crt file from elastic-stack-ca.p12, which I've created using this documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html.

I used elastic-stack-ca.p12 to sign all keys that I have created for all ELK nodes. I've created key and cert for filebeat and signed with elastic-stack-ca.p12 also.

For Filebeat i've also created key and crt using this elastic-stack-ca.p12.

What is my mistake?

2

The beats protocol is not HTTP based so I do not think curl will work.

3

but it says:

Connected to logstash-ip-address (logstash-ip-address) port 5044 (#0)

When i execute:

curl -v --cacert ./certs/ca.crt logstash_ip_address:5044

I get

* About to connect() to logstash_ip_address port 5044 (#0)
*   Trying logstash_ip_address...
* Connected to logstash_ip_address (logstash_ip_address) port 5044 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: logstash_ip_address:5044
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

Also the documentation says:

"You can use curl to validate the certificate even though the protocol used to communicate with Logstash is not based on HTTP"