We use a AWS NLB in front of our ES cluster and are receiving errors in the logstash logs stating that Elasticsearch is unreachable. The full error is below:
2018-09-18_19:00:10.19502 [ERROR] 2018-09-18 19:00:10.194 [[main]>worker0] elasticsearch - Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://elasticsearch.login.gov.internal:9200/][Manticore::SocketException] Connection reset", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
2018-09-18_19:00:12.19657 [ERROR] 2018-09-18 19:00:12.196 [[main]>worker0] elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
It seems to be a result of the connection being reset by the NLB or something else. I'm pretty sure we can modify the connection reset value but I'm also wondering if using a LB in front of the ES nodes is the best way to go. It seems like Elasticsearch handles much of what a LB does internally making our NLB redundant if not problematic. For instance, using the classic load balancer resulted in large IO wait on the logstash instance. We thought we resolved the issue by moving to a NLB but we're still getting the above errors. Can anyone shed more light on this topic?