Logstash losing connection to elastic intermittently

I am running Elastic stack on Docker swarm with xpac monitoring enabled. Everything is running fine with the cluster status as green.

Data Nodes : 4
Ingest Nodes : 4
logstash nodes : 4

but when I tail the logs for logststash, I keep getting these message very regularly complaining about Host Unreachable

[WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection reset {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection reset", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

[ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection reset", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}

[INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}

[WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}

I have checked the Xpac monitoring from kibana and everything seems to be fine. All the containers are running fine with no restarts. If i curl to elastic from the logstash container, it always returns a response. I dont find any way to find why this error is coming up.

I dont see similar errors in kibana, heartbeat or apm-server logs. If someone can help identify the source of this problem it would be very helpful.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.