The logstash keep getting 'UNEXPECTED POOL ERROR', even the Elastic Search is in GREEN mode

Elastic Stack Logstash
1
  • Version: Logstash 5.5, AWS Elasticsearch 5.5
  • Operating System: Linux CentOS 7(Logstash)
  • Sample Error:

[2018-05-01T12:05:52,235][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://XXX.amazonaws.com:443/][Manticore::ClientProtocolException] XXX.amazonaws.com:443 failed to respond {:url=>https://XXX.amazonaws.com:443/, :error_message=>"Elasticsearch Unreachable: [https://XXX.amazonaws.com:443/][Manticore::ClientProtocolException] XXX.amazonaws.com:443 failed to respond", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

[2018-05-01T12:05:52,235][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://XXX.amazonaws.com:443/][Manticore::ClientProtocolException] XXX.amazonaws.com:443 failed to respond", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}

[2018-05-01T12:05:54,088][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}

[2018-05-01T12:05:55,920][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://XXXX.amazonaws.com:443/, :path=>"/"}

[2018-05-01T12:05:55,936][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://XXX.amazonaws.com:443/"}

  • Thread_Pool Settings:
    bulk:"max" : 4, "min":4, "queue_size": 200
2

It looks like the machine on which your Logstash is running is losing its route to your Elasticsearch host(s).

3

Thanks for your reply. So you think that may be caused by network glitch?

4

Whether it is a network glitch, a change in your virtual network configuration, or the hosts you are pointing at being replaced by new nodes with new IPs, there are many possibilities.

5

Is there a way that I can narrow down the reasons? What could be best solution to this kind of issue.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.