Elastiscsearch Output - Attempted to send a bulk request

[Gochou]

Hi,

So from time to time I get this error in my logstash logs:

[2019-02-01T01:35:18,762][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [host:443/][Manticore::ClientProtocolException] host:443 failed to respond", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2019-02-01T01:35:20,771][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
[2019-02-01T01:35:22,626][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"host:443/"}
[2019-02-01T03:11:02,517][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [host:443/][Manticore::ClientProtocolException] host:443 failed to respond {:url=>host:443/, :error_message=>"Elasticsearch Unreachable: [host:443/][Manticore::ClientProtocolException] host:443 failed to respond", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2019-02-01T03:11:02,517][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [host:443/][Manticore::ClientProtocolException] host:443 failed to respond", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2019-02-01T03:11:02,704][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"host:443/"}

It doesn't actually seem to affect my Elastic Stack (no missing logs in ES) but I would still like to resolve it if possible.
I'm running Filebeat 6.5.4, Logstash 6.5.4 and Elasticsearch AWS 5.6.8.
I know they aren't supposed to be compatible if you look a the compatibility matrix but it's working fine for me.

Do you have any idea what's causing this ?

Thanks for your help.

[guyboertje]

It is probably caused when AWS ES is too busy to accept a connection. The attempt is queued, retried and succeeds.

You might consider tuning your AWS ES for better performance.

[Gochou]

I don't have access to the AWS ES tuning, so if it retries and succeeds is it an error my cluster can live with temporarily without too many issues ? (For now I'm just doing testing so I'll see to tune it when I move on to something more serious)
Or, if it is actually dangerous to keep, will tuning down Logstash help ?

Thanks again for your help.