Low throupught

[sbampa]

Hi all,
i have a problem in my ES cluster:

from 5 days, all my filebeat instances receive an i/o exception when sending log to logstash instances.

Logstash logs, write several times at minutes this messagge:

[2019-02-08T11:49:32,368][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elastic:xxxxxx@127.0.0.1:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://elastic:xxxxxx@127.0.0.1:9200/, :error_message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@127.0.0.1:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2019-02-08T11:49:32,369][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@127.0.0.1:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2019-02-08T11:49:37,235][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@127.0.0.1:9200/, :path=>"/"}
[2019-02-08T11:49:37,238][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@127.0.0.1:9200/"}
[2019-02-08T11:49:46,965][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elastic:xxxxxx@172.16.19.223:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://elastic:xxxxxx@172.16.19.223:9200/, :error_message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@172.16.19.223:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2019-02-08T11:49:46,966][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@172.16.19.223:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2019-02-08T11:49:47,242][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@172.16.19.223:9200/, :path=>"/"}
[2019-02-08T11:49:47,245][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@172.16.19.223:9200/"}

I can't find the problem...network is not busy, resources is good on all components (Beat, logstash and Elasticsearch).
Can someone help me??

[juka (Jürgen)]

That looks like networking issues. HostUnreachableError, running a health check and then Restored connection to ES. There are two ips involved, did you manually check the network connections are good. Maybe a tcpdump on the according ports whether any data is actually flowing?

Your logfile says that logstash has problems sending data to elasticsearch. I don't know how this exactly relates to your filebeat having i/o exceptions. You should check your logstash to elasticsearch data flow. What does your output section in logstash conf looks like? Do you send data to two ES instances?

Oh, and are you running on cloud or hardware?

Good Luck!