Hi guys,
i've set up security for ELK. Everythings works, except the communication between Filebeat and Logstash. Everything is version 7.4
filebeat.yml
output.logstash:
# The Logstash hosts
hosts: ["ip-address:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
ssl.certificate_authorities: ["/certs/ca.crt"]
# Certificate for SSL client authentication
ssl.certificate: "/certs/filebeat_client.crt"
# Client Certificate Key
ssl.key: "/filebeat_client.key"
ssl.key_passphrase: password
logstash config file:
input {
beats {
port => "5044"
ssl => true
ssl_certificate_authorities => ["/certs/logstash-01/ca.crt"]
ssl_certificate => "/certs/logstash-01/logstash-01.crt"
ssl_key => "/certs/logstash-01/logstash01.pkcs8.key"
ssl_verify_mode => "peer"
}
}
filter {
csv {
separator => ","
columns => ["id","insert_time","unix_time"]
convert => {
"id" => "integer"
}
}
}
output {
elasticsearch {
hosts => ["elastic01.com:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
ssl => true
ssl_certificate_verification => true
cacert => '/certs/logstash-01/elastic-ca.pem'
user => logstash_internal
password => password
}
}
When I try to start filebeat, I see following error in the filebeat logs:
|2019-12-17T16:01:20.284Z|INFO|pipeline/output.go:95|Connecting to backoff(async(tcp://logstash-ip-address:5044))|
|---|---|---|---|
|2019-12-17T16:01:21.434Z|ERROR|pipeline/output.go:100|Failed to connect to backoff(async(tcp://logstash-ip-address:5044)): read tcp filebeat-machine-ip-addres:54108->logstash-ip-address:5044: read: connection reset by peer|
following in the logstash logs:
[2019-12-17T16:01:20,304][INFO ][org.logstash.beats.BeatsHandler][main] [local: 0.0.0.0:5044, remote: undefined] Handling exception: javax.net.ssl.SSLException: java.lang.Exception: Error setting private key (error:0b000074:X.509 certificate routines:OPENSSL_internal:KEY_VALUES_MISMATCH)
[2019-12-17T16:01:20,304][INFO ][org.logstash.beats.BeatsHandler][main] [local: 0.0.0.0:5044, remote: undefined] Handling exception: javax.net.ssl.SSLException: java.lang.Exception: Error setting private key (error:0b000074:X.509 certificate routines:OPENSSL_internal:KEY_VALUES_MISMATCH)
[2019-12-17T16:01:20,305][WARN ][io.netty.channel.DefaultChannelPipeline][main] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: java.lang.Exception: Error setting private key (error:0b000074:X.509 certificate routines:OPENSSL_internal:KEY_VALUES_MISMATCH)
I tried to:
curl -v --cacert ./certs/ca.crt https://logstash-ip-address:5044
and I received following:
* About to connect() to logstash-ip-address port 5044 (#0)
* Trying logstash-ip-address...
* Connected to logstash-ip-address (logstash-ip-address) port 5044 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /certs/ca.crt
CApath: none
* NSS error -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
* Closing connection 0
curl: (35) TCP connection reset by peer
Info:
I've extracted the ca.crt file from elastic-stack-ca.p12, which I've created using this documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html.
I used elastic-stack-ca.p12 to sign all keys that I have created for all ELK nodes. I've created key and cert for filebeat and signed with elastic-stack-ca.p12 also.
For Filebeat i've also created key and crt using this elastic-stack-ca.p12.
What is my mistake?