Securing Elastic Stack: Logstash

Elastic Stack Elasticsearch
1

I have been working on securing my elastic stack, and am now stuck on getting Logstash to communicate with Elasticsearch securely.

I have a cluster with three nodes (call them elk1, elk2, and elk3), each running an Elasticsearch data node that is an eligible master, logstash, and kibana.

Here is what I did, so far...

Converted the general PK12 certificate that I had created when first securing communication between Elasticsearch nodes:

openssl pkcs12 -in elk-certificates.p12 -out /etc/logstash/logstash.pem -clcerts -nokeys

And also obtained the CA into a CRT file:

openssl pkcs12 -in elk-certificates.p12 -cacerts -nokeys -chain | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > logstash-ca.crt

I added the info to the Logstash output filters:

input {
  beats {
    port => 5044
  }
}
output {
  if [fields][log_for] {
    elasticsearch {
      ssl => true
      ssl_certificate_verification => true
      cacert => '/etc/logstash/logstash.pem'
      hosts => [ "elk1:9200", "elk2:9200", "elk3:9200" ]
      user => "logstash_local"
      password => "REDACTED"
      index => "logstash-%{[fields][log_for]}-%{+YYYY.MM.dd}"
    }
  }
}

And added the info to the Logstash config file:

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_local
xpack.monitoring.elasticsearch.password: REDACTED
xpack.monitoring.elasticsearch.hosts: [ "https://elk1:9200", "https://elk2:9200", "https://elk3:9200" ]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/logstash-ca.crt
xpack.monitoring.elasticsearch.sniffing: true
xpack.monitoring.collection.interval: 10s
xpack.monitoring.collection.pipeline.details.enabled: true

I am getting these error from Logstash (10.xxx.xxx.xx1 is the IP address of the elk1 server, etc):

[2021-10-14T17:25:28,057][ERROR][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash][7cd6dd12d7cf61636b45ef3b3af9abd08fb1c71c5e7c263521eb573db99a98c4] Encountered a retryable error. Will Retry with exponential backoff  {:code=>403, :url=>"https://10.xxx.xxx.xx1:9200/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s"}
[2021-10-14T17:25:35,375][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Host name '10.xxx.xxx.xx1' does not match the certificate subject provided by the peer (CN=instance)"}
[2021-10-14T17:26:05,376][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Host name '10.xxx.xxx.xx2' does not match the certificate subject provided by the peer (CN=instance)"}
[2021-10-14T17:26:32,069][ERROR][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash][7cd6dd12d7cf61636b45ef3b3af9abd08fb1c71c5e7c263521eb573db99a98c4] Encountered a retryable error. Will Retry with exponential backoff  {:code=>403, :url=>"https://xxx.xxx.xx3:9200/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s"}
[2021-10-14T17:26:35,376][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Host name '10.xxx.xxx.xx3' does not match the certificate subject provided by the peer (CN=instance)"}
[2021-10-14T17:27:05,376][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Host name '10.xxx.xxx.xx3' does not match the certificate subject provided by the peer (CN=instance)"}

I am hoping someone can help me figure out what to look at next for debugging.

2

The error is related to Logstash unable to verify the name provided by Elastic (The IP really) against the one present in the certificate (assume to be "elk1"); is logstash able to perform dns reverse resolution ? (Spawn a shell and try this:

host 10.xxx.xxx.xxx3

I'm not really sure if reverse dns is going to solve your problem, maybe the hostname config in elastic is the IP address, so, the nodes present with it instead of dns name, leading to this error.

As a last option I'm pretty sure that Logstash offers some configurations for ignoring ssl errors & trust any certificate.

3

Please don't create multiple topics on the same question, it makes it harder to help you :slight_smile: