I am running Elastic stack on Docker swarm with xpac monitoring enabled. Everything is running fine with the cluster status as green.
Data Nodes : 4
Ingest Nodes : 4
logstash nodes : 4
but when I tail the logs for logststash, I keep getting these message very regularly complaining about Host Unreachable
[WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection reset {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection reset", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection reset", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
[WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
I have checked the Xpac monitoring from kibana and everything seems to be fine. All the containers are running fine with no restarts. If i curl to elastic from the logstash container, it always returns a response. I dont find any way to find why this error is coming up.
I dont see similar errors in kibana, heartbeat or apm-server logs. If someone can help identify the source of this problem it would be very helpful.