We have a very busy ES instance with 200+ servers streaming via logstash. At peak volume times, we are seeing these "dead ES instance" errors when logstash attempts to connect to the instance:
[2021-05-12T14:49:46,293][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://log_writer:xxxxxx@es.company.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://log_writer:xxxxxx@es.company.com:443/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
This occurs during high volume and several of the logstash nodes throw these errors. It's interesting that the error mentions "unable to find valid certification path", but we confirmed that the certs are working as expected.
Any ideas or recommendations on how to handle these "dead ES instance" errors from logstash?