How do people typically handle shard failures in their results?


#1

If I understand correctly, we can get an OK response from elastic (ie no
error) but if there are shard failures in the response, it potentially
means that results are incomplete/incorrect. From my observation, we can
get failures on all shards - and elastic still returns OK (which was a bit
surprising to me)

What kinds of approaches to people typically use to deal with shard
failures?

For my application, if there are shard failures, essentially my results are
inaccurate/incorrect - so I need to return an error to the client.
Returning bad results is worse than returning an error.

I am inclined to turn any shard failure into an exception.
Is this quite common? Does it make sense to add a feature to the elastic
api ? (ie request.setTreatShardFailuresAsErrors(true)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/461fa217-d664-47e9-a60d-88ea9506327d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Dealing with node failures
(Itamar Syn-Hershko) #2

Every man and his own business requirements and SLAs. Many scenarios I've
seen would consider two-thirds of shards responding as an OK scenario and
not even show a notice about results being partial. But of course it makes
sense to throw an exception if your business requires / expects that.

FWIW, this is usually why you use replicas, so even if a shard goes down
there's a back-up shard (ideally more than one) that you can fallback to.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Wed, Jun 18, 2014 at 12:45 PM, mooky nick.minutello@gmail.com wrote:

If I understand correctly, we can get an OK response from elastic (ie no
error) but if there are shard failures in the response, it potentially
means that results are incomplete/incorrect. From my observation, we can
get failures on all shards - and elastic still returns OK (which was a bit
surprising to me)

What kinds of approaches to people typically use to deal with shard
failures?

For my application, if there are shard failures, essentially my results
are inaccurate/incorrect - so I need to return an error to the client.
Returning bad results is worse than returning an error.

I am inclined to turn any shard failure into an exception.
Is this quite common? Does it make sense to add a feature to the elastic
api ? (ie request.setTreatShardFailuresAsErrors(true)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/461fa217-d664-47e9-a60d-88ea9506327d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/461fa217-d664-47e9-a60d-88ea9506327d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zu7ECzf7bA_YG0XoGaVrPr1CUyKzhq1VRn4XD47CH%2B0MQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Shay Banon) #3

If it fails on the primary shard, then a failure is returned. If it worked, and a replica failed, then that replica is deemed a failed replica, and will get allocated somewhere else in the cluster. Maybe an example of where a failure on "all" shards would help here?

On Jun 18, 2014, at 11:45, mooky nick.minutello@gmail.com wrote:

If I understand correctly, we can get an OK response from elastic (ie no error) but if there are shard failures in the response, it potentially means that results are incomplete/incorrect. From my observation, we can get failures on all shards - and elastic still returns OK (which was a bit surprising to me)

What kinds of approaches to people typically use to deal with shard failures?

For my application, if there are shard failures, essentially my results are inaccurate/incorrect - so I need to return an error to the client. Returning bad results is worse than returning an error.

I am inclined to turn any shard failure into an exception.
Is this quite common? Does it make sense to add a feature to the elastic api ? (ie request.setTreatShardFailuresAsErrors(true)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/461fa217-d664-47e9-a60d-88ea9506327d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/FFDC7083-24CB-484D-B337-65582596D555%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Nik Everett) #4

On Fri, Jun 20, 2014 at 7:08 AM, Shay Banon kimchy@gmail.com wrote:

If it fails on the primary shard, then a failure is returned. If it
worked, and a replica failed, then that replica is deemed a failed replica,
and will get allocated somewhere else in the cluster. Maybe an example of
where a failure on “all” shards would help here?

I think its more about searches and they can fail on one shard but not
other for all sorts of reasons. Queue full, unfortunate script, bug, only
one shard had results and the query asked for something weird like to use
the postings highlighter when postings aren't stored. Lots of reasons.

I log the event and move on. I toyed with outputting a warning to the user
but didn't have time to implement it. We're pretty diligent with our logs
so we'd notice the log and run it down.

If the failure is caused by the queue being full only on one node, we'd
likely notice that real quick as ganglia would lose it. This happened to
me recently when we put a node without an ssd into a cluster with ssds. It
couldn't keep up and dropped a ton of searches. In our defense, we didn't
know the rest of the cluster had ssds so we were double surprised.

Nik

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Shay Banon) #5

Ahh, I see. If its related to searches, then yea, the search response includes details about the total shards that the search was executed on, the successful shards, and failed shards. They are important to check to understand if one gets partial results.

In the REST API, if there is a total "failure", then it will return the "worst" status code out of all the shards in the response. In the Java API, the search response will be returned (with no exception), so the content of the search has to be checked (which is a good practice anyhow). It might make sense to raise an exception in the Java API if all shards failed, I am on the fence on this one, since anyhow a check needs to be performed on the result.

On Jun 20, 2014, at 13:22, Nikolas Everett nik9000@gmail.com wrote:

On Fri, Jun 20, 2014 at 7:08 AM, Shay Banon kimchy@gmail.com wrote:
If it fails on the primary shard, then a failure is returned. If it worked, and a replica failed, then that replica is deemed a failed replica, and will get allocated somewhere else in the cluster. Maybe an example of where a failure on "all" shards would help here?

I think its more about searches and they can fail on one shard but not other for all sorts of reasons. Queue full, unfortunate script, bug, only one shard had results and the query asked for something weird like to use the postings highlighter when postings aren't stored. Lots of reasons.

I log the event and move on. I toyed with outputting a warning to the user but didn't have time to implement it. We're pretty diligent with our logs so we'd notice the log and run it down.

If the failure is caused by the queue being full only on one node, we'd likely notice that real quick as ganglia would lose it. This happened to me recently when we put a node without an ssd into a cluster with ssds. It couldn't keep up and dropped a ton of searches. In our defense, we didn't know the rest of the cluster had ssds so we were double surprised.

Nik

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9A9246D5-338B-4B93-B2FD-4D3B93F621F2%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


#6

I see. Thanks.
I wasnt expecting to have to check for shard failures.
Would it make sense to have a config or request setting whereby any shard
failures are reported as exceptions (java api).
In the space where we have been using elastic, partial results are bad. We
would end up having to always write code to turn shard failures into
exceptions....

-Nick

On Friday, 20 June 2014 13:26:35 UTC+1, kimchy wrote:

Ahh, I see. If its related to searches, then yea, the search response
includes details about the total shards that the search was executed on,
the successful shards, and failed shards. They are important to check to
understand if one gets partial results.

In the REST API, if there is a total “failure”, then it will return the
“worst” status code out of all the shards in the response. In the Java API,
the search response will be returned (with no exception), so the content of
the search has to be checked (which is a good practice anyhow). It might
make sense to raise an exception in the Java API if all shards failed, I am
on the fence on this one, since anyhow a check needs to be performed on the
result.

On Jun 20, 2014, at 13:22, Nikolas Everett <nik...@gmail.com <javascript:>>
wrote:

On Fri, Jun 20, 2014 at 7:08 AM, Shay Banon <kim...@gmail.com
<javascript:>> wrote:

If it fails on the primary shard, then a failure is returned. If it
worked, and a replica failed, then that replica is deemed a failed replica,
and will get allocated somewhere else in the cluster. Maybe an example of
where a failure on “all” shards would help here?

I think its more about searches and they can fail on one shard but not
other for all sorts of reasons. Queue full, unfortunate script, bug, only
one shard had results and the query asked for something weird like to use
the postings highlighter when postings aren't stored. Lots of reasons.

I log the event and move on. I toyed with outputting a warning to the
user but didn't have time to implement it. We're pretty diligent with our
logs so we'd notice the log and run it down.

If the failure is caused by the queue being full only on one node, we'd
likely notice that real quick as ganglia would lose it. This happened to
me recently when we put a node without an ssd into a cluster with ssds. It
couldn't keep up and dropped a ton of searches. In our defense, we didn't
know the rest of the cluster had ssds so we were double surprised.

Nik

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2dvNM-wu%3Due4trJzAtLV%3Dz1xK0MVNxhYkUKv2g68z3VQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/63f3b5a8-02b2-4a2f-bb47-b42b46ff4d54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #7