Logstash to Elasticsearch connectivity issue after configuring SSL

[noodler_1096 (Askar)]

As a part of enabling encryption on services, I tried the following tasks.

1. Enable SSL on ElasticSearch
2. Enable SSL on Kibana
3. Enable SSL for the communication channel between Kibana and Elasticsearch.
   Above things were done and tested fine.

As I had logstash as well part of my stack, I wanted to enable SSL on the communication channel between Logstash and Elasticsearch.

For this, I referred Elasticsearch documentation and had the changes as below for logstash outputs:
output {

  if [type] == "log" {
      stdout {
        codec => rubydebug
      }
      elasticsearch {
        hosts => ["elasticsearch"]
        index => "logstash-logs-%{+YYYY.MM.dd}"
        ssl => true
        ssl_certificate_verification => false
        cacert => "/config/domain.crt"
        action => "index"
        manage_template => false
    }
  } else if [type] == "beats" {
      stdout {
        codec => rubydebug
      }
      elasticsearch {
        hosts => ["elasticsearch"]
        index => "filebeat-%{+YYYY.MM.dd}"
        ssl => true
        ssl_certificate_verification => false
        cacert => "/config/domain.crt"
        action => "index"
        manage_template => false
    }
  }
}

After having the above configuration, below is the error I am getting:
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties

[2019-03-19T12:34:23,408][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}

[2019-03-19T12:34:23,420][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}

[2019-03-19T12:34:23,892][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

[2019-03-19T12:34:23,901][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}

[2019-03-19T12:34:23,935][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"a8a70965-acc3-47f1-8557-335b3771fc90", :path=>"/usr/share/logstash/data/uuid"}

[2019-03-19T12:34:26,018][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@elasticsearch:9200/]}}

[2019-03-19T12:34:26,408][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

[2019-03-19T12:34:26,525][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://elastic:xxxxxx@elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:url=>https://elastic:xxxxxx@elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

[2019-03-19T12:34:26,537][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

[2019-03-19T12:34:26,581][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.

It was working without SSL configuration:
output {

  if [type] == "log" {
      stdout {
        codec => rubydebug
      }
      elasticsearch {
        hosts => ["elasticsearch"]
        index => "logstash-logs-%{+YYYY.MM.dd}"
        user => "${ELASTICSEARCH_USERNAME}"
        password => "${ELASTICSEARCH_PASSWORD}"
    }
  } else if [type] == "beats" {
      stdout {
        codec => rubydebug
      }
      elasticsearch {
        hosts => ["elasticsearch"]
        index => "filebeat-%{+YYYY.MM.dd}"
        user => "${ELASTICSEARCH_USERNAME}"
        password => "${ELASTICSEARCH_PASSWORD}"
    }
  }

}

[dadoonet (David Pilato)]

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

[noodler_1096 (Askar)]

Formatting code done.