We have ELK stack on Kubernetes(version 7.0.0), Logstash is failing to interact with elasticsearch with the following warning and errors, all other components like filebeat and kibana are well functioning.
[2021-05-11T11:45:21,872][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://xxxx:xxxxxx@elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://xxxx:xxxxxx@elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
[2021-05-11T11:45:22,042][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://xxxx:xxxxxx@elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:url=>https://osher:xxxxxx@elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [https://xxxx:xxxxxx@elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2021-05-11T11:45:22,053][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [https://xxxx:xxxxxx@elasticsearch:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
The elaticsearch output plugin from logstash.conf:
elasticsearch {
hosts => "${ELASTICSEARCH_HOST}"
user => "${ELASTICSEARCH_USERNAME}"
password => "${ELASTICSEARCH_PASSWORD}"
ssl => true
cacert => "/usr/share/logstash/config/certs/elasticsearch-ca.pem"
ilm_rollover_alias => "development"
ilm_pattern => "000001"
ilm_policy => "dev-policy"
}
Will appreciate your assistance,