# Logstash-output-elasticsearch load balancing not working when one of the nodes is down

**URL:** https://discuss.elastic.co/t/logstash-output-elasticsearch-load-balancing-not-working-when-one-of-the-nodes-is-down/287300
**Category:** Logstash
**Created:** [October 21, 2021, 10:09am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-load-balancing-not-working-when-one-of-the-nodes-is-down/287300 "2021-10-21T10:09:22Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![preetish\_P](https://avatars.discourse-cdn.com/v4/letter/p/77aa72/32.png) [@preetish\_P](https://discuss.elastic.co/u/preetish_P)
#### Post date: [October 21, 2021, 10:09am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-load-balancing-not-working-when-one-of-the-nodes-is-down/287300/1 "2021-10-21T10:09:22Z")

</div>

Hi Team,

Currently we are working on negative testing of Elasticsearch multi-node clustering. Out current setup is:  
node 1: Elasticsearch,logstash,kibana  
node 2: Elasticsearch  
node 3: Elasticsearch

the logstash on node 1 is pointing to all 3 nodes as below:

```auto
elasticsearch {
                    hosts => ["${ES_NODE_1}","${ES_NODE_2}","${ES_NODE_3}"]                    
                    index => "<index-name>"           
                    user => "${ES_USER_NAME}"
                    password => "${ES_USER_PASSWORD}"
                    ssl => true
                    cacert => "${ES_CERT_AUTH}"
                }

```

We have around 20 pipelines with all kinds of inputs (lumberjack, http\_poller, jdbc).

We are observing that when one of the ES nodes is brought down, the pipelines having http\_poller stop working (we call set of 3 APIs every minute). The ones based on lumberjack continue to work.

We continuously get this error in logstash-plain.log, which is expected:

```auto
[2021-10-19T17:43:02,646][WARN][logstash.outputs.elasticsearch][<pipeline>] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://ES-NODE-1:9201/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://ES-NODE-1:9201/][Manticore::SocketException] Connection refused (Connection refused)"}
[2021-10-19T17:43:02,642][WARN][logstash.outputs.elasticsearch][<module>] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://es-node-1:9201/][Manticore::SocketException] Connection refused (Connection refused) {:url=>https://es-node-1:9201/, :error_message=>"Elasticsearch Unreachable: [https://es-node-1:9201/][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

```

We guessed that http\_poller inputs might be getting starved of OS socket connections as all of them are being used up for internal healthcheck. We used below settings:

```auto
/etc/security/limits.conf:

logstash soft nofile 65536 
logstash hard nofile 65536

```

```auto
   resurrect_delay => 300
   retry_max_interval => 8

```

but still no luck.

Has anyone observed this behaviour ? is there any setting which we can assign to logstash output plugin to ensure it continues to work with one node down ?

Thanks

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 18, 2021, 10:09am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-load-balancing-not-working-when-one-of-the-nodes-is-down/287300/2 "2021-11-18T10:09:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.