Logstash to Elasticsearch - SSL Connection Error

[Paul_14]

Hello,

I am attempting to enable SSL encryption between Logstash and Elasticsearch but on restarting my logstash node, I receive error:

[2019-07-03T12:36:42,992][DEBUG][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_writer:xxxxxx@espdata1:9200/, :path=>"/"}
[2019-07-03T12:36:43,003][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_writer:xxxxxx@espdata1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'https://espdata1:9200/'"}
[2019-07-03T12:36:43,003][DEBUG][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_writer:xxxxxx@espdata2:9200/, :path=>"/"}
[2019-07-03T12:36:43,012][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_writer:xxxxxx@espdata2:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_writer:xxxxxx@espdata2:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

I have enabled HTTP encryption in my three Elasticsearch nodes as per elasticsearch.yml snippet below. My SSL certificates are generated by an internal CA.

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.key: config/espdata0.key
xpack.security.transport.ssl.certificate: config/espdata0.crt
xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/config/CARoot.crt","/etc/elasticsearch/config/CAIntermediate.crt"]

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: config/espdata0.key
xpack.security.http.ssl.certificate: config/espdata0.local.crt
xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/config/CARoot.crt","/etc/elasticsearch/config/CAIntermediate.crt"]

In Logstash, I have configured the below in my .conf file output filter. I set certificate verification to 'false' after reading some other posts related to issues with certificates that were generated from an internal CA... In the cacert field I also tried '/etc/logstash/config/CARoot.crt' but had the same outcome.

output {
elasticsearch {

             hosts => ["espdata0:9200","espdata1:9200","espdata2"]
             index => "meterflow-%{+YYYY.MM.dd}"
             user => "logstash_writer"
             password => "Elastic$tack"
             ssl => true
             ssl_certificate_verification => false
             cacert => "/etc/logstash/config/CAIntermediate.crt"
                   }
   }

In logstash.yml I have set monitoring to false:

xpack.monitoring.enabled: false

Can anyone suggest where I may be going wrong here?

Thanks,
Paul

[ikakavas (Ioannis Kakavas)]

It looks like there is an issue with your logstash_writer user and not with TLS.

Please share some additional information. Can you reach all espdata0, espdata1, espdata2 nodes from the machine where logstash is running? Did you follow this documentation Have you created this user in elasticsearch with the appropriate roles ?

Can you verify the password is correct ? i.e. try the authenticate API

curl -u logstash_writer 'https://espdata2:9200/_security/_authenticate'

and share the output with us.

[Paul_14]

Hi Ikakavas,

First off, there was a mistake on my part. My user is logstash_internal - which has the role logstash_writer. So I tried your instruction with logstash_internal.

> curl -u logstash_internal 'https://espdata0:9200/_security/_authenticate'
> Enter host password for user 'logstash_internal':
> curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
> More details here: http://curl.haxx.se/docs/sslcerts.html
> 
> curl performs SSL certificate verification by default, using a "bundle"
>  of Certificate Authority (CA) public keys (CA certs). If the default
>  bundle file isn't adequate, you can specify an alternate file
>  using the --cacert option.
> If this HTTPS server uses a certificate signed by a CA represented in
>  the bundle, the certificate verification probably failed due to a
>  problem with the certificate (it might be expired, or the name might
>  not match the domain name in the URL).
> If you'd like to turn off curl's verification of the certificate, use
>  the -k (or --insecure) option.

I then tried it with the --cacert argument with gave a healthier response:

curl  --cacert CAIntermediate.crt -u logstash_internal 'https://espdata0:9200/_security/_authenticate'
Enter host password for user 'logstash_internal':
{"username":"logstash_internal","roles":["logstash_writer"],"full_name":"Internal Logstash User","email":null,"metadata":{},"enabled":true,"authentication_realm":{"name":"default_native","type":"native"},"lookup_realm":{"name":"default_native","type":"native"}}root@esp-logstash:/etc/logstash/config#

I also updated my logstash output filter to use the logstash_internal user... Upon restarting, this resolved the issue. I was spending a lot of time looking into SSL configuration and not the native user authentication.

Thanks again for you reply.