shardFailures in SearchResponse showing from previous query


(Lar Mader) #1

I am confused about how to interpret the SearchResponse.shardFailures
array, as this is showing failures from a previous failed query in the
SearchResponse from a new successful query, as follows:

  1. First execute a query that fails, either due to malformed json in the
    query, or with correct json but a query that is invalid.
  2. Then execute a new query that is correct and succeeds. The
    SearchResponse from this query has failedShards == 0, but the shardFailures
    array has items from the previous failed query.

So is this the correct behavior??
And if so, how does one know which shardFailures are from the current query
or a previous one. This becomes more of an issue I think when two queries
fail in a row - in this case the SearchResponse from the second query has
items in the shardFailures array from both queries.

Lar


(Jörg Prante) #2

Hi,

are you using replica? The shard failures happen on all shards, but a
response is generated by draining only from one replica set of shard.
Clearing the exceptions will not occur on the other replica shard set. So
the messages live on and they happen to get visible by a follow-up query,
when shard addressing changes to some other replica set.

Try if the effect disappears when removing replica. I think it's a harmless
glitch. Clients should check for HTTP status code for errors, or failed
shard count, but not by examining shard failure exception messages in the
first place.

Best regards,

Jörg

On Friday, August 3, 2012 1:06:47 AM UTC+2, lmader wrote:

I am confused about how to interpret the SearchResponse.shardFailures
array, as this is showing failures from a previous failed query in the
SearchResponse from a new successful query, as follows:

  1. First execute a query that fails, either due to malformed json in the
    query, or with correct json but a query that is invalid.
  2. Then execute a new query that is correct and succeeds. The
    SearchResponse from this query has failedShards == 0, but the shardFailures
    array has items from the previous failed query.

So is this the correct behavior??
And if so, how does one know which shardFailures are from the current
query or a previous one. This becomes more of an issue I think when two
queries fail in a row - in this case the SearchResponse from the second
query has items in the shardFailures array from both queries.

Lar


(Lar Mader) #3

Well, I am not using replica sets.

So it seems to me that this is a bug, although perhaps it would be
considered minor. It seems incorrect for the SearchResponse to have
exception messages in the shardFailures array when the shards didn't fail
(in the result from the second query). This makes the shardFailures
messages misleading and unreliable.

Or am I still missing something?

Thanks!
Lar

On Fri, Aug 3, 2012 at 4:39 PM, Jörg Prante joergprante@gmail.com wrote:

Hi,

are you using replica? The shard failures happen on all shards, but a
response is generated by draining only from one replica set of shard.
Clearing the exceptions will not occur on the other replica shard set. So
the messages live on and they happen to get visible by a follow-up query,
when shard addressing changes to some other replica set.

Try if the effect disappears when removing replica. I think it's a
harmless glitch. Clients should check for HTTP status code for errors, or
failed shard count, but not by examining shard failure exception messages
in the first place.

Best regards,

Jörg

On Friday, August 3, 2012 1:06:47 AM UTC+2, lmader wrote:

I am confused about how to interpret the SearchResponse.**shardFailures
array, as this is showing failures from a previous failed query in the
SearchResponse from a new successful query, as follows:

  1. First execute a query that fails, either due to malformed json in the
    query, or with correct json but a query that is invalid.
  2. Then execute a new query that is correct and succeeds. The
    SearchResponse from this query has failedShards == 0, but the shardFailures
    array has items from the previous failed query.

So is this the correct behavior??
And if so, how does one know which shardFailures are from the current
query or a previous one. This becomes more of an issue I think when two
queries fail in a row - in this case the SearchResponse from the second
query has items in the shardFailures array from both queries.

Lar


(Shay Banon) #4

If this is the case, its definitely a bug, can you maybe open an issue with a gist (curl) recreation? which version are you using?

On Aug 6, 2012, at 6:28 PM, Lar Mader lmaderintrepid@gmail.com wrote:

Well, I am not using replica sets.

So it seems to me that this is a bug, although perhaps it would be considered minor. It seems incorrect for the SearchResponse to have exception messages in the shardFailures array when the shards didn't fail (in the result from the second query). This makes the shardFailures messages misleading and unreliable.

Or am I still missing something?

Thanks!
Lar

On Fri, Aug 3, 2012 at 4:39 PM, Jörg Prante joergprante@gmail.com wrote:
Hi,

are you using replica? The shard failures happen on all shards, but a response is generated by draining only from one replica set of shard. Clearing the exceptions will not occur on the other replica shard set. So the messages live on and they happen to get visible by a follow-up query, when shard addressing changes to some other replica set.

Try if the effect disappears when removing replica. I think it's a harmless glitch. Clients should check for HTTP status code for errors, or failed shard count, but not by examining shard failure exception messages in the first place.

Best regards,

Jörg

On Friday, August 3, 2012 1:06:47 AM UTC+2, lmader wrote:
I am confused about how to interpret the SearchResponse.shardFailures array, as this is showing failures from a previous failed query in the SearchResponse from a new successful query, as follows:

  1. First execute a query that fails, either due to malformed json in the query, or with correct json but a query that is invalid.
  2. Then execute a new query that is correct and succeeds. The SearchResponse from this query has failedShards == 0, but the shardFailures array has items from the previous failed query.

So is this the correct behavior??
And if so, how does one know which shardFailures are from the current query or a previous one. This becomes more of an issue I think when two queries fail in a row - in this case the SearchResponse from the second query has items in the shardFailures array from both queries.

Lar


(Lar Mader) #5

Ok, I opened an issue with a set of steps, as curl commands at:

I am using elasticsearch version 0.18.7.

Cheers,
Lar

On Tue, Aug 7, 2012 at 2:31 PM, Shay Banon kimchy@gmail.com wrote:

If this is the case, its definitely a bug, can you maybe open an issue
with a gist (curl) recreation? which version are you using?

On Aug 6, 2012, at 6:28 PM, Lar Mader lmaderintrepid@gmail.com wrote:

Well, I am not using replica sets.

So it seems to me that this is a bug, although perhaps it would be
considered minor. It seems incorrect for the SearchResponse to have
exception messages in the shardFailures array when the shards didn't fail
(in the result from the second query). This makes the shardFailures
messages misleading and unreliable.

Or am I still missing something?

Thanks!
Lar

On Fri, Aug 3, 2012 at 4:39 PM, Jörg Prante joergprante@gmail.com wrote:

Hi,

are you using replica? The shard failures happen on all shards, but a
response is generated by draining only from one replica set of shard.
Clearing the exceptions will not occur on the other replica shard set. So
the messages live on and they happen to get visible by a follow-up query,
when shard addressing changes to some other replica set.

Try if the effect disappears when removing replica. I think it's a
harmless glitch. Clients should check for HTTP status code for errors, or
failed shard count, but not by examining shard failure exception messages
in the first place.

Best regards,

Jörg

On Friday, August 3, 2012 1:06:47 AM UTC+2, lmader wrote:

I am confused about how to interpret the SearchResponse.**shardFailures
array, as this is showing failures from a previous failed query in the
SearchResponse from a new successful query, as follows:

  1. First execute a query that fails, either due to malformed json in the
    query, or with correct json but a query that is invalid.
  2. Then execute a new query that is correct and succeeds. The
    SearchResponse from this query has failedShards == 0, but the shardFailures
    array has items from the previous failed query.

So is this the correct behavior??
And if so, how does one know which shardFailures are from the current
query or a previous one. This becomes more of an issue I think when two
queries fail in a row - in this case the SearchResponse from the second
query has items in the shardFailures array from both queries.

Lar


(Shay Banon) #6

This has been fixed in 0.19 (I just double checked your test case just in case).

On Aug 8, 2012, at 1:34 AM, Lar Mader lmaderintrepid@gmail.com wrote:

Ok, I opened an issue with a set of steps, as curl commands at:

https://github.com/elasticsearch/elasticsearch/issues/2148

I am using elasticsearch version 0.18.7.

Cheers,
Lar

On Tue, Aug 7, 2012 at 2:31 PM, Shay Banon kimchy@gmail.com wrote:
If this is the case, its definitely a bug, can you maybe open an issue with a gist (curl) recreation? which version are you using?

On Aug 6, 2012, at 6:28 PM, Lar Mader lmaderintrepid@gmail.com wrote:

Well, I am not using replica sets.

So it seems to me that this is a bug, although perhaps it would be considered minor. It seems incorrect for the SearchResponse to have exception messages in the shardFailures array when the shards didn't fail (in the result from the second query). This makes the shardFailures messages misleading and unreliable.

Or am I still missing something?

Thanks!
Lar

On Fri, Aug 3, 2012 at 4:39 PM, Jörg Prante joergprante@gmail.com wrote:
Hi,

are you using replica? The shard failures happen on all shards, but a response is generated by draining only from one replica set of shard. Clearing the exceptions will not occur on the other replica shard set. So the messages live on and they happen to get visible by a follow-up query, when shard addressing changes to some other replica set.

Try if the effect disappears when removing replica. I think it's a harmless glitch. Clients should check for HTTP status code for errors, or failed shard count, but not by examining shard failure exception messages in the first place.

Best regards,

Jörg

On Friday, August 3, 2012 1:06:47 AM UTC+2, lmader wrote:
I am confused about how to interpret the SearchResponse.shardFailures array, as this is showing failures from a previous failed query in the SearchResponse from a new successful query, as follows:

  1. First execute a query that fails, either due to malformed json in the query, or with correct json but a query that is invalid.
  2. Then execute a new query that is correct and succeeds. The SearchResponse from this query has failedShards == 0, but the shardFailures array has items from the previous failed query.

So is this the correct behavior??
And if so, how does one know which shardFailures are from the current query or a previous one. This becomes more of an issue I think when two queries fail in a row - in this case the SearchResponse from the second query has items in the shardFailures array from both queries.

Lar


(Lar Mader) #7

Ah, you are correct sir, this is indeed fixed in 0.19. Thanks!!
Lar

On Wed, Aug 8, 2012 at 2:49 AM, Shay Banon kimchy@gmail.com wrote:

This has been fixed in 0.19 (I just double checked your test case just in
case).

On Aug 8, 2012, at 1:34 AM, Lar Mader lmaderintrepid@gmail.com wrote:

Ok, I opened an issue with a set of steps, as curl commands at:

https://github.com/elasticsearch/elasticsearch/issues/2148

I am using elasticsearch version 0.18.7.

Cheers,
Lar

On Tue, Aug 7, 2012 at 2:31 PM, Shay Banon kimchy@gmail.com wrote:

If this is the case, its definitely a bug, can you maybe open an issue
with a gist (curl) recreation? which version are you using?

On Aug 6, 2012, at 6:28 PM, Lar Mader lmaderintrepid@gmail.com wrote:

Well, I am not using replica sets.

So it seems to me that this is a bug, although perhaps it would be
considered minor. It seems incorrect for the SearchResponse to have
exception messages in the shardFailures array when the shards didn't fail
(in the result from the second query). This makes the shardFailures
messages misleading and unreliable.

Or am I still missing something?

Thanks!
Lar

On Fri, Aug 3, 2012 at 4:39 PM, Jörg Prante joergprante@gmail.comwrote:

Hi,

are you using replica? The shard failures happen on all shards, but a
response is generated by draining only from one replica set of shard.
Clearing the exceptions will not occur on the other replica shard set. So
the messages live on and they happen to get visible by a follow-up query,
when shard addressing changes to some other replica set.

Try if the effect disappears when removing replica. I think it's a
harmless glitch. Clients should check for HTTP status code for errors, or
failed shard count, but not by examining shard failure exception messages
in the first place.

Best regards,

Jörg

On Friday, August 3, 2012 1:06:47 AM UTC+2, lmader wrote:

I am confused about how to interpret the SearchResponse.**shardFailures
array, as this is showing failures from a previous failed query in the
SearchResponse from a new successful query, as follows:

  1. First execute a query that fails, either due to malformed json in
    the query, or with correct json but a query that is invalid.
  2. Then execute a new query that is correct and succeeds. The
    SearchResponse from this query has failedShards == 0, but the shardFailures
    array has items from the previous failed query.

So is this the correct behavior??
And if so, how does one know which shardFailures are from the current
query or a previous one. This becomes more of an issue I think when two
queries fail in a row - in this case the SearchResponse from the second
query has items in the shardFailures array from both queries.

Lar


(system) #8