Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down?

[Yazid_Abed_Alqader]

Hi
I have these error messages in logstash logs:

[2023-10-15T08:01:31,446][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [htt
ps://***.us-east-1.es.amazonaws.com:443/][Manticore::SocketTimeout] Read timed out {:url=>https://***.us
-east-1.es.amazonaws.com:443/, :error_message=>"Elasticsearch Unreachable: [https://***.us-east-1.es.amazonaws.com:443/][Manticore::SocketTimeout] Read ti
med out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}                                                                                                         
[2023-10-15T08:01:31,446][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://***.us-east-1.es.amazonaws.com:443/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::
Pool::HostUnreachableError", :will_retry_in_seconds=>2}                                                                                                                                                     
[2023-10-15T08:01:32,221][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://***.us-east-1.es.amazonaws.com:443/"}

everything was working fine and suddenly these messages started appearing in the logs.
I'm using the AWS OpenSearch service, and its status is green and it's accessible from the losstash pods.
I have checked this thread and tried to use amazon_es plugin and I got the same error message,

[system]

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

[Yazid_Abed_Alqader]

The issue is caused by overstressing the ES server, there was a heavy performers test and it created hundreds of millions of logs.
The weird thing is the status of the ES was green!!

[leandrojmp (Leandro Pereira)]

@Yazid_Abed_Alqader

The weird thing is the status of the ES was green!!

The status is unrelated to your issue, the elasticsearch cluster status depends only on the availability of the shards, if all primary and replica shards are available, then the status will be green.