# Deleted items appears when searching a replica shard **URL:** \ **Category:** Elasticsearch\ **Created:** [January 31, 2013, 4:26am UTC](https://discuss.elastic.co/t/deleted-items-appears-when-searching-a-replica-shard/10564 "2013-01-31T04:26:09Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [January 31, 2013, 4:26am UTC](https://discuss.elastic.co/t/deleted-items-appears-when-searching-a-replica-shard/10564/1 "2013-01-31T04:26:09Z")
We're using ES 0.19.2 at the moment, in a 3-node configuration. I have an odd case I can't explain. We have an item in an ES index that comes back, depending on what host/shard/replica receives the query. We tracked down this issue to a particular ID for a given type, and when one does a simple GET request against this ID, ala : curl -XGET [http://localhost:9200/au1/task/168802590](http://localhost:9200/au1/task/168802590) we get alternating-ish views, some showing it doesn't exist, and some showing it does: { \_index: au1 \_type: task \_id: 168802590 exists: false } then do it again, and we get something like this (redacted): { \_index: au1 \_type: task \_id: 168802590 \_version: 1359072320783 exists: true \_source: { id: 168802590 type: awaitingreview description: BLAH BLAH creationDate: 1359068463727 projectId: 30979 projectName: BLAH BLAH stepName: BLAH BLAH assignedBy: BLAH BLAH mailId: null dueDate: 1359154862937 referenceLabel: BLAH BLAH aggregateCount: 1 featureId: 2 startDate: null aggregateId: 134736434 assignedToUser: BLAH BLAH aggregated: true pertinent: true assignedByUser: BLAH BLAH assignedByUserId: 415818 assignedToUserId: 415818 dueDateIso: 20130126 } } If I then modify this get to use the ...?preference=\_primary to hit only the primary shard, I get 100% consistent results that exists=false (as in, the item is deleted on the primary shard). If I then change preference=\_local, and walk over the each node guessing which node might hold the replica for this item (I don't know what the routing value would be for this, so don't know which shard it would go to?) I get consistent results with the full results above. I can't seem to prove which shard replica that is affected here....? This leads me to think that the replica for a given shard does not have a correct view of the deleted items for each segment. I think I have 2 options: - use \_optimize to properly expunge the deletes (maybe with just the expunge deletes only option, though I still have to use max\_num\_segments=1 to force the optimize to happen. - Set Replicas to 0, and then rebuild the replica set again by setting it back to 1. This is not my preferred, given the small-ish window of vulnerability of having no redundancy. Before I do any of the above, is there any other ideas out there on what I can, or things I can do to collect more info to help work out why? Is there other ways to force removal of deleted items? cheers, Paul Smith -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [January 31, 2013, 9:13am UTC](https://discuss.elastic.co/t/deleted-items-appears-when-searching-a-replica-shard/10564/2 "2013-01-31T09:13:00Z")
Hi Paul > This leads me to think that the replica for a given shard does not > have a correct view of the deleted items for each segment. Try using the index-status API to compare doc counts between shards: curl -XGET '[http://127.0.0.1:9200/\_status?pretty=1](http://127.0.0.1:9200/_status?pretty=1)' Also, worth trying: 1. Flush the indices, wait for a bit and check again: 2. Clear the caches, wait for 60 seconds and check again: 3. Try redeleting the doc, or possibly recreating then deleting If none of these options work, then the below > I think I have 2 options: > > - use \_optimize to properly expunge the deletes (maybe with just the > expunge deletes only option, though I still have to use > max\_num\_segments=1 to force the optimize to happen. and optimize can be IO heavy > - Set Replicas to 0, and then rebuild the replica set again by setting > it back to 1. This is not my preferred, given the small-ish window of > vulnerability of having no redundancy. This would be my preferred choice. Disable allocation before and reenable after. Recovery should be quick for all shards, but possibly slightly slower for the incorrect shard. As to why it has happened? I don't know. Have you had OOMs or some other similar disruption on the network? clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [February 6, 2013, 11:53am UTC](https://discuss.elastic.co/t/deleted-items-appears-when-searching-a-replica-shard/10564/3 "2013-02-06T11:53:54Z")
I tried the flush, cache clear and re-deleting and re-putting back, no avail. Rather than go down the replica=0 approach first, I tried optimizing down to max\_num\_segments=1 during idle time, but the result still comes up if it goes through the replica shard. using preference=\_primary always comes out as deleted ok, so I stilly suspect the shard replica, though by optimizing I'm not sure why the replica never received a 'cleansed' copy of the bad segment. I'm down to setting replicas to 0 now, which is a bit nail biting, Murphy being the sod that he is, this is when stuff happens. Alternatively I could do a rolling restart of each node, slowly, to all the shards to rebalance, but I since I don't know which shard contains this record, I may end up getting the replica copy instead of the primary, but it's possible that then it can be properly deleted after that. Is there a way to work out by hand which shard an item will go to by it's ID? What's the default Hashing algorithm? Dunno if Shay or anyone else has either ideas? (reminder we're on 0.19.2 at the moment, and yes, don't I wish I could upgrade). regards, Paul On 31 January 2013 20:13, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote: > Hi Paul > > > This leads me to think that the replica for a given shard does not > > have a correct view of the deleted items for each segment. > > Try using the index-status API to compare doc counts between shards: > curl -XGET '[http://127.0.0.1:9200/\_status?pretty=1](http://127.0.0.1:9200/_status?pretty=1)' > > Also, worth trying: > > 1. Flush the indices, wait for a bit and check again: > > 2. Clear the caches, wait for 60 seconds and check again: > > 3. Try redeleting the doc, or possibly recreating then deleting > > If none of these options work, then the below > > > I think I have 2 options: > > > > - use \_optimize to properly expunge the deletes (maybe with just the > > expunge deletes only option, though I still have to use > > max\_num\_segments=1 to force the optimize to happen. > > and optimize can be IO heavy > > > - Set Replicas to 0, and then rebuild the replica set again by setting > > it back to 1. This is not my preferred, given the small-ish window of > > vulnerability of having no redundancy. > > This would be my preferred choice. Disable allocation before and > reenable after. Recovery should be quick for all shards, but possibly > slightly slower for the incorrect shard. > > As to why it has happened? I don't know. Have you had OOMs or some > other similar disruption on the network? > > clint > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 6, 2013, 12:10pm UTC](https://discuss.elastic.co/t/deleted-items-appears-when-searching-a-replica-shard/10564/4 "2013-02-06T12:10:41Z")
Hi Paul > Rather than go down the replica=0 approach first, I tried optimizing > down to max\_num\_segments=1 during idle time, but the result still > comes up if it goes through the replica shard. using > preference=\_primary always comes out as deleted ok, so I stilly > suspect the shard replica, though by optimizing I'm not sure why the > replica never received a 'cleansed' copy of the bad segment. optimize happens local to each shard, so it wouldn't trigger a "get-latest-from-primary" action. > I'm down to setting replicas to 0 now, which is a bit nail biting, > Murphy being the sod that he is, this is when stuff happens. > Alternatively I could do a rolling restart of each node, slowly, to > all the shards to rebalance, but I since I don't know which shard > contains this record, I may end up getting the replica copy instead of > the primary, but it's possible that then it can be properly deleted > after that. Is there a way to work out by hand which shard an item > will go to by it's ID? What's the default Hashing algorithm? Do an 'ids' search with explain turned on. That'll return the node and shard: curl -XGET '[http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1)' -d ' { "fields" : , "query" : { "ids" : { "values" : [ "wVaNiUKTRE-Ax5LLR61OkA" ] } }, "explain" : 1 } ' # { # "hits" : { # "hits" : [ # { # "\_score" : 1, # "\_index" : "test", # "\_shard" : 2, # "\_id" : "wVaNiUKTRE-Ax5LLR61OkA", # "\_node" : "x8E-zFIsTGSXFo3xq4hb4A", # "\_type" : "test", # "\_explanation" : { # "value" : 1, # "details" : [ # { # "value" : 1, # "description" : "boost" # }, # { # "value" : 1, # "description" : "queryNorm" # } # ], # "description" : "ConstantScore(\_uid:test#wVaNiUKTRE-Ax5LLR61OkA), product of:" # } # } # ], # "max\_score" : 1, # "total" : 1 # }, # "timed\_out" : false, # "\_shards" : { # "failed" : 0, # "successful" : 5, # "total" : 5 # }, # "took" : 2 # } clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [February 6, 2013, 9:00pm UTC](https://discuss.elastic.co/t/deleted-items-appears-when-searching-a-replica-shard/10564/5 "2013-02-06T21:00:19Z")
Thanks for that tip Clinton! That's identified the host with the shard replica. If I run the above type of query I get flapping results (no results, then a result from the host containing the replica shard), and I can see it's the host with the replica of that shard that is always returning that result. My next step will be to shutdown ES on that node, let the cluster re-replicate the shard to one of the other nodes and see if the problem goes away, before bringing that node backup again. I'll report back on what I find. thanks again, Paul On 6 February 2013 23:10, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote: > Hi Paul > > > Rather than go down the replica=0 approach first, I tried optimizing > > down to max\_num\_segments=1 during idle time, but the result still > > comes up if it goes through the replica shard. using > > preference=\_primary always comes out as deleted ok, so I stilly > > suspect the shard replica, though by optimizing I'm not sure why the > > replica never received a 'cleansed' copy of the bad segment. > > optimize happens local to each shard, so it wouldn't trigger a > "get-latest-from-primary" action. > > > I'm down to setting replicas to 0 now, which is a bit nail biting, > > Murphy being the sod that he is, this is when stuff happens. > > Alternatively I could do a rolling restart of each node, slowly, to > > all the shards to rebalance, but I since I don't know which shard > > contains this record, I may end up getting the replica copy instead of > > the primary, but it's possible that then it can be properly deleted > > after that. Is there a way to work out by hand which shard an item > > will go to by it's ID? What's the default Hashing algorithm? > > Do an 'ids' search with explain turned on. That'll return the node and > shard: > > curl -XGET '[http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1)' -d ' > { > "fields" : , > "query" : { > "ids" : { > "values" : [ > "wVaNiUKTRE-Ax5LLR61OkA" > ] > } > }, > "explain" : 1 > } > ' > > # { > > # "hits" : { > > # "hits" : [ > > # { > > # "\_score" : 1, > > # "\_index" : "test", > > # "\_shard" : 2, > > # "\_id" : "wVaNiUKTRE-Ax5LLR61OkA", > > # "\_node" : "x8E-zFIsTGSXFo3xq4hb4A", > > # "\_type" : "test", > > # "\_explanation" : { > > # "value" : 1, > > # "details" : [ > > # { > > # "value" : 1, > > # "description" : "boost" > > # }, > > # { > > # "value" : 1, > > # "description" : "queryNorm" > > # } > > # ], > > # "description" : > > "ConstantScore(\_uid:test#wVaNiUKTRE-Ax5LLR61OkA), product of:" > > # } > > # } > > # ], > > # "max\_score" : 1, > > # "total" : 1 > > # }, > > # "timed\_out" : false, > > # "\_shards" : { > > # "failed" : 0, > > # "successful" : 5, > > # "total" : 5 > > # }, > > # "took" : 2 > > # } > > clint > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [February 7, 2013, 9:59am UTC](https://discuss.elastic.co/t/deleted-items-appears-when-searching-a-replica-shard/10564/6 "2013-02-07T09:59:34Z")
Thanks to Clinton's tip of finding the shard info for this ID I worked out which node help the replica shard, and gracefully shut it down, let the cluster replicate and rebalance (because I don't have options to prevent allocation in 0.19.2) and then brought the node up again (and re-balanced). Now the dead item no longer appears to come back in results, so I think this dodgy replica shard is now history, though what happened to it is of course, unclear. But my problem now appears fixed (touch wood). Thanks Clinton! Paul On 7 February 2013 08:00, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote: > Thanks for that tip Clinton! That's identified the host with the shard > replica. If I run the above type of query I get flapping results (no > results, then a result from the host containing the replica shard), and I > can see it's the host with the replica of that shard that is always > returning that result. > > My next step will be to shutdown ES on that node, let the cluster > re-replicate the shard to one of the other nodes and see if the problem > goes away, before bringing that node backup again. > > I'll report back on what I find. > > thanks again, > > Paul > > On 6 February 2013 23:10, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote: > > > Hi Paul > > > > > Rather than go down the replica=0 approach first, I tried optimizing > > > down to max\_num\_segments=1 during idle time, but the result still > > > comes up if it goes through the replica shard. using > > > preference=\_primary always comes out as deleted ok, so I stilly > > > suspect the shard replica, though by optimizing I'm not sure why the > > > replica never received a 'cleansed' copy of the bad segment. > > > > optimize happens local to each shard, so it wouldn't trigger a > > "get-latest-from-primary" action. > > > > > I'm down to setting replicas to 0 now, which is a bit nail biting, > > > Murphy being the sod that he is, this is when stuff happens. > > > Alternatively I could do a rolling restart of each node, slowly, to > > > all the shards to rebalance, but I since I don't know which shard > > > contains this record, I may end up getting the replica copy instead of > > > the primary, but it's possible that then it can be properly deleted > > > after that. Is there a way to work out by hand which shard an item > > > will go to by it's ID? What's the default Hashing algorithm? > > > > Do an 'ids' search with explain turned on. That'll return the node and > > shard: > > > > curl -XGET '[http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1)' -d ' > > { > > "fields" : , > > "query" : { > > "ids" : { > > "values" : [ > > "wVaNiUKTRE-Ax5LLR61OkA" > > ] > > } > > }, > > "explain" : 1 > > } > > ' > > > > # { > > > > # "hits" : { > > > > # "hits" : [ > > > > # { > > > > # "\_score" : 1, > > > > # "\_index" : "test", > > > > # "\_shard" : 2, > > > > # "\_id" : "wVaNiUKTRE-Ax5LLR61OkA", > > > > # "\_node" : "x8E-zFIsTGSXFo3xq4hb4A", > > > > # "\_type" : "test", > > > > # "\_explanation" : { > > > > # "value" : 1, > > > > # "details" : [ > > > > # { > > > > # "value" : 1, > > > > # "description" : "boost" > > > > # }, > > > > # { > > > > # "value" : 1, > > > > # "description" : "queryNorm" > > > > # } > > > > # ], > > > > # "description" : > > > > "ConstantScore(\_uid:test#wVaNiUKTRE-Ax5LLR61OkA), product of:" > > > > # } > > > > # } > > > > # ], > > > > # "max\_score" : 1, > > > > # "total" : 1 > > > > # }, > > > > # "timed\_out" : false, > > > > # "\_shards" : { > > > > # "failed" : 0, > > > > # "successful" : 5, > > > > # "total" : 5 > > > > # }, > > > > # "took" : 2 > > > > # } > > > > clint > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:52am UTC](https://discuss.elastic.co/t/deleted-items-appears-when-searching-a-replica-shard/10564/7 "2017-07-06T02:52:36Z")