# Dealing with node failures

**URL:** <https://discuss.elastic.co/t/dealing-with-node-failures/127931>\
**Category:** Elasticsearch\
**Created:** [April 13, 2018, 8:33am UTC](https://discuss.elastic.co/t/dealing-with-node-failures/127931 "2018-04-13T08:33:52Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dd\_d](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@dd\_d](https://discuss.elastic.co/u/dd_d)\
**Post date:** [April 13, 2018, 8:33am UTC](https://discuss.elastic.co/t/dealing-with-node-failures/127931/1 "2018-04-13T08:33:52Z")

</div>

My cluster config :  
ES version : 6.2.3  
number of nodes : 3 (master eligible/data)  
number of replica : 1

I ran tests sending concurrent search requests to ES cluster using java rest client api(3 hosts, with sniffer), then killed a node.  
My expectation was that,  
since there are replicas on the other two nodes, the requests would succeed.  
But it failed partially and the response status was 200.

```
{

```

"took": 558,  
"timed\_out": false,  
"\_shards": {  
"total": 6,  
"successful": 5,  
"skipped": 0,  
"failed": 1,  
"failures": [  
{  
"shard": 5,  
"index": "v1",  
"reason": {  
"type": "node\_disconnected\_exception",  
"reason": "~~[indices:data/read/search[phase/fetch/id]] disconnected"  
}  
}  
]  
}  
}

So my questions are,

1. Is there any way to get success response(without shard failure)?

2. If I have to deal with the retry on that cases, how long it would be take to get success response?  
When I retried right after the shard failures, all requests ended with success. I wonder If I can get success response always with single retry on the node\_disconnected\_exception failure.

Thanks.

Releated topics :  
Should Elasticsearch return a non-200 response if there are shard failures? #18978

> <https://github.com/elastic/elasticsearch/issues/18978>
>
> Lots of Elasticsearch tasks are forked onto a bunch of shards. When those shards… fail, Elasticsearch returns the failures in the json at \`\_shards.failures\` including the HTTP response code that that failure deserves \_but\_ it will return an HTTP 200 response code so long as a single shard succeeds. I think it should return a non-200 HTTP status if any shard fails.
> 
> Do we want to:
> 1. Continue as we are and return a 200 code if any shard succeeds.
> 2. Return a non-200 code if any shards fail. We'd return the highest numbered failure because that is what we do now if \_all\_ shards fail. You can get each response code in the \`failures\` array in the response. We could also talk about \[RFC 4918\](https://tools.ietf.org/html/rfc4918)'s \`207 Multi-Status\` but at first glance that specifies some XML response we aren't going to implement.
> 3. Add a boolean to the request to toggle between the two behaviors. We'd have to pick a default but we could default to the old behavior for 5.0 if we didn't want the change to be breaking.

How do people typically handle shard failures in their results?

> [@How do people typically handle shard failures in their results?](https://discuss.elastic.co/t/how-do-people-typically-handle-shard-failures-in-their-results/18176):
>
> If I understand correctly, we can get an OK response from elastic (ie no error) but if there are shard failures in the response, it potentially means that results are incomplete/incorrect. From my observation, we can get failures on all shards - and elastic still returns OK (which was a bit surprising to me) What kinds of approaches to people typically use to deal with shard failures? For my application, if there are shard failures, essentially my results are inaccurate/incorrect - so I …

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 11, 2018, 8:33am UTC](https://discuss.elastic.co/t/dealing-with-node-failures/127931/2 "2018-05-11T08:33:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.