# The indexing or search request send to down node

**URL:** <https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022>\
**Category:** Elasticsearch\
**Created:** [July 24, 2023, 12:52am UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022 "2023-07-24T00:52:12Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chimu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chimu/32/48956_2.png) [@Chimu](https://discuss.elastic.co/u/Chimu)\
**Post date:** [July 24, 2023, 12:52am UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/1 "2023-07-24T00:52:12Z")

</div>

I have an Elasticsearch (v5.6.10) cluster with 3 nodes.

- Node A : Master
- Node B : Master + Data
- Node C : Master + Data

There are 6 shards per data node with replication set as 1. All 6 primary nodes are in Node B and all 6 replicas are in Node C.

When I shutdown one node for maintenance I can see indexing or search requests still trying to reach that node and failing. I suspect this is because the client connecting to elastic is configured with all three node IPs.

Is there any way to avoid the requests reaching that down node?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 24, 2023, 5:56am UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/2 "2023-07-24T05:56:33Z")

</div>

I guess it depends on the client.

You must definitely upgrade everything. The cluster, the client... It's a way too old.

---

<div class="post-metadata">

**Author:** ![Chimu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chimu/32/48956_2.png) [@Chimu](https://discuss.elastic.co/u/Chimu)\
**Post date:** [July 24, 2023, 9:38pm UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/3 "2023-07-24T21:38:26Z")

</div>

Can you please elaborate on that?

I am trying to find a solution where client does not face errors in this situation.

I saw that cluster status being yellow does not cause any issue during the maintenance period. But if a request goes to the down node, that’s when the problem occurs.

And, yes they are very old. I am planning for the upgrade, but it will take some time. For now I need to keep things running.. 🙂

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 24, 2023, 11:41pm UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/4 "2023-07-24T23:41:20Z")

</div>

Which client are you using?

---

<div class="post-metadata">

**Author:** ![Chimu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chimu/32/48956_2.png) [@Chimu](https://discuss.elastic.co/u/Chimu)\
**Post date:** [July 25, 2023, 1:09am UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/5 "2023-07-25T01:09:41Z")

</div>

It’s a java client - “elasticsearch-rest-high-level-client” and all the elastic node ips are provided as a list while creating the rest client.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 25, 2023, 9:00am UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/6 "2023-07-25T09:00:30Z")

</div>

Did you add the sniffer? [Sniffer | Elasticsearch Java API Client [8.8] | Elastic](https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/sniffer.html)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 25, 2023, 9:07am UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/7 "2023-07-25T09:07:27Z")

</div>

The client contains a connection pool, which should mark connections as down once this has been detected. You could therefore see a few requests target the downed node before this is detected. This assumes you are using tbe client correctlt as a singleton and not creating it for each request.

---

<div class="post-metadata">

**Author:** ![Chimu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chimu/32/48956_2.png) [@Chimu](https://discuss.elastic.co/u/Chimu)\
**Post date:** [July 25, 2023, 9:42pm UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/8 "2023-07-25T21:42:09Z")

</div>

Yes, I did. I was also looking into that. It seems I missed something.

Just to confirm, sniffer will remove any down node from the active node list and also add it back once it is up?

---

<div class="post-metadata">

**Author:** ![Chimu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chimu/32/48956_2.png) [@Chimu](https://discuss.elastic.co/u/Chimu)\
**Post date:** [July 25, 2023, 9:46pm UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/9 "2023-07-25T21:46:39Z")

</div>

Yes, it is implemented as a singleton, but the connection to the down node is not automatically withdrawn from the connection pool.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 26, 2023, 7:34am UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/10 "2023-07-26T07:34:15Z")

</div>

I never used it but I guess it works that way according to the doc 😉

> It is also possible to enable sniffing on failure, meaning that after each failure the nodes list gets updated straightaway rather than at the following ordinary sniffing round. In this case a `SniffOnFailureListener` needs to be created at first and provided at `RestClient` creation. Also once the `Sniffer` is later created, it needs to be associated with that same `SniffOnFailureListener` instance, which will be notified at each failure and use the `Sniffer` to perform the additional sniffing round as described.

---

<div class="post-metadata">

**Author:** ![Chimu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chimu/32/48956_2.png) [@Chimu](https://discuss.elastic.co/u/Chimu)\
**Post date:** [July 27, 2023, 3:35am UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/11 "2023-07-27T03:35:03Z")

</div>

I hope so too. I will give it a try and update here. Thanks a lot for all the guidance.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 24, 2023, 3:35am UTC](https://discuss.elastic.co/t/the-indexing-or-search-request-send-to-down-node/339022/12 "2023-08-24T03:35:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
