# Primary vs. replica shard inconsistencies? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 30, 2014, 10:28pm UTC](https://discuss.elastic.co/t/primary-vs-replica-shard-inconsistencies/15514 "2014-01-30T22:28:26Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![xavier1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xavier1/32/1836_2.png) [@xavier1](https://discuss.elastic.co/u/xavier1)\ **Post date:** [January 30, 2014, 10:28pm UTC](https://discuss.elastic.co/t/primary-vs-replica-shard-inconsistencies/15514/1 "2014-01-30T22:28:26Z")
We're currently running ElasticSearch 0.90.5. When doing the same search across different query heads, I'm seeing an inconsistent number of results. However, if I add preference=\_primary (or \_primary\_first) I get the same results across the board. I have checked that all query heads report the same nodes in the cluster. This makes me think that the replica shards are not consistent with the primary shards. This index is not actively being written to, so I executed a flush and refresh manually. That didn't seem to change anything. Anyone have any ideas on what this could be? Is this possibly a bug? Thanks in advance. -Xavier -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/82ae0c25-9603-401d-ad04-f26287d630f8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/82ae0c25-9603-401d-ad04-f26287d630f8%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Binh\_Ly](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@Binh\_Ly](https://discuss.elastic.co/u/Binh_Ly)\ **Post date:** [January 30, 2014, 10:44pm UTC](https://discuss.elastic.co/t/primary-vs-replica-shard-inconsistencies/15514/2 "2014-01-30T22:44:25Z")
Xavier, can you post an example of 1 full query and then also show how the results of this one query is inconsistent? Just trying to understand what is inconsistent. Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/038735ba-cef6-4634-9d46-7ff39dffc4d2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/038735ba-cef6-4634-9d46-7ff39dffc4d2%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [January 31, 2014, 12:00am UTC](https://discuss.elastic.co/t/primary-vs-replica-shard-inconsistencies/15514/3 "2014-01-31T00:00:49Z")
If you can narrow down a specific few IDs of results that appear/disappear based on the primary/replica shard, and confirm through an explicit GET of that ID with the preference=\_local on the primary shard & replica for that result. To work out which shard # a specific ID belongs to, you can run this query: curl -XGET '_[http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1) [http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1)_' -d ' { "fields" : , "query" : { "ids" : { "values" : [ "123456789" ] } }, "explain" : 1 } ' where the "values" attribute you place the ID of the item you're after. Within the result response you'l see the shard Id, use that to identify which host is the primary and which is the replica. You can then run the GET query with the preference=\_local on each of those hosts and see if the primary or replica shows the result. You will need to understand whether the item that is 'flappy' (appearing/disappearing depending on the shard being searched) is supposed to be in there or not, perhaps checking the data store that is the source of the index (is it a dB?). We have very infrequent case where the replica shard is not properly receiving a delete at least with 0.19.10. The delete successfully applies to the Primary, but the Replica still holds the value and returns it within search results. We have loads of insert/update/delete activity and the number of flappy items is very small, but it is definitely a thing. see this previous thread: [http://elasticsearch-users.115913.n3.nabble.com/Deleted-items-appears-when-searching-a-replica-shard-td4029075.html](http://elasticsearch-users.115913.n3.nabble.com/Deleted-items-appears-when-searching-a-replica-shard-td4029075.html) If it is the replica shard that's incorrect (my bet), the way to fix it is to relocate the replica shard to another host. The relocation will take the copy of the primary (correct copy) and recreate a new replica shard, effectively neutralizing the inconsistency. We have written a tool, Scrutineer ([https://github.com/aconex/scrutineer](https://github.com/aconex/scrutineer)) which can help detect inconsistencies in your cluster. I also have a tool not yet published to github that can help check these Primary/Replica inconsistencies if that would help (you pass a list of IDs to it and it'll check whether they're flappy between the primary & replica or not). It can also help automate the rebuilding of just the replica shards by shunting them around (rather than a full rolling restart of ALL the shards, just the shard replicas you want) cheers, Paul Smith On 31 January 2014 09:44, Binh Ly [binh@hibalo.com](mailto:binh@hibalo.com) wrote: > Xavier, can you post an example of 1 full query and then also show how the > results of this one query is inconsistent? Just trying to understand what > is inconsistent. Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/038735ba-cef6-4634-9d46-7ff39dffc4d2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/038735ba-cef6-4634-9d46-7ff39dffc4d2%40googlegroups.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHfYWB51LsR5XH6G5VX3KcGdPU8mVUc-eEiROPS1wjwQGkaobg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHfYWB51LsR5XH6G5VX3KcGdPU8mVUc-eEiROPS1wjwQGkaobg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![xavier1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xavier1/32/1836_2.png) [@xavier1](https://discuss.elastic.co/u/xavier1)\ **Post date:** [January 31, 2014, 12:45am UTC](https://discuss.elastic.co/t/primary-vs-replica-shard-inconsistencies/15514/4 "2014-01-31T00:45:50Z")
We have 4 query heads total (esq1.r6, esq2.r6, esq3.r7, esq4.r7). Interestingly query heads in the same rack give the same results. We don't do deletes at all on these indices so that shouldn't be an issue. Unfortunately at the moment I can't do preference=\_local while getting the \_id(s) directly because we don't allow access on 9200 on our worker nodes. I might be able to right some code to figure this out though. Either way here's my id results from the different heads. esq2.r6 gets 28 total results esq3.r7 gets 9 total results $curl -XGET "[http://esq2.r6:9200/events/\_search?q=sessionId:1390953880&size=100](http://esq2.r6:9200/events/_search?q=sessionId:1390953880&size=100)" | jq '.hits.hits.\_id' | sort % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 19337 100 19337 0 0 1039k 0 --:--:-- --:--:-- --:--:-- 1049k "0LcI\_px4SZy5ZQkI\_V7Qyw" "1sAGREtMSfK8OIxZErm8RQ" "6IV2v4TFTr-Gl1eC6hrj0Q" "6nwMexTHQBmFxfykOgKqWA" "7hFYs6y-QG6wGYEkoBKmdg" "9MTM10SeQ2yqWIb08oPnFA" "aELtGN6DQpmdRlQbr8i0uA" "AUHUg6k0QZOf\_oGjsjSsGA" "Bo\_u1eYGSF2LeU78kbcFZg" "EWs1K8YsR9-IBSAWK6ld7A" "Fx4l6\_axSGCxpyFm7C7BSQ" "gpCrAZrNTNezWPfensER3g" "HAFmGcWuQAylxGjmnZZkSQ" "HB4Kwz3RSWWH5NHvyH4JMg" "H-eP-33FREOtq7v0uBPWbQ" "\_IH6W4DoTRmdms0FJNlg4g" "iK\_3TbzcSj2-MbMXip\_XFg" "J4bjPFIcQ1ewrQqjN2qz6Q" "kfonMDBuR--UIhkyM2cWrg" "Kr6-9-3uR9Wp2923n-O2NA" "Nw\_9rjwvQ62u-HsuWIm53A" "QRmY8R2MQemuePb0EkYxWA" "usloSJzQRzCpOQ8bxKi2vA" "w9NGEWg-QiivMpjyurYKrA" "wKy-YzB-TK2lnK86Sx2RBA" "y2ZmJ-\_GRAmi3eHy1y8jzw" "ZmFj7w4hR5Cvy-owCLmZ1Q" "ZmlndPBLT-ivuOxm\_A7yDA" $curl -XGET "[http://esq3.r7:9200/events/\_search?q=sessionId:1390953880&size=100](http://esq3.r7:9200/events/_search?q=sessionId:1390953880&size=100)" | jq '.hits.hits.\_id' | sort % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6808 100 6808 0 0 70082 0 --:--:-- --:--:-- --:--:-- 70185 "1sAGREtMSfK8OIxZErm8RQ" "7hFYs6y-QG6wGYEkoBKmdg" "aELtGN6DQpmdRlQbr8i0uA" "Fx4l6\_axSGCxpyFm7C7BSQ" "HAFmGcWuQAylxGjmnZZkSQ" "H-eP-33FREOtq7v0uBPWbQ" "QRmY8R2MQemuePb0EkYxWA" "wKy-YzB-TK2lnK86Sx2RBA" "y2ZmJ-\_GRAmi3eHy1y8jzw" And here is es3.r7 with preference=\_primary\_first: $curl -XGET "[http://esq3.r7/events/\_search?q=sessionId:1390953880&size=100&preference=\_primary\_first](http://esq3.r7/events/_search?q=sessionId:1390953880&size=100&preference=_primary_first)" | jq '.hits.hits.\_id' | sort % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 19335 100 19335 0 0 871k 0 --:--:-- --:--:-- --:--:-- 899k "0LcI\_px4SZy5ZQkI\_V7Qyw" "1sAGREtMSfK8OIxZErm8RQ" "6IV2v4TFTr-Gl1eC6hrj0Q" "6nwMexTHQBmFxfykOgKqWA" "7hFYs6y-QG6wGYEkoBKmdg" "9MTM10SeQ2yqWIb08oPnFA" "aELtGN6DQpmdRlQbr8i0uA" "AUHUg6k0QZOf\_oGjsjSsGA" "Bo\_u1eYGSF2LeU78kbcFZg" "EWs1K8YsR9-IBSAWK6ld7A" "Fx4l6\_axSGCxpyFm7C7BSQ" "gpCrAZrNTNezWPfensER3g" "HAFmGcWuQAylxGjmnZZkSQ" "HB4Kwz3RSWWH5NHvyH4JMg" "H-eP-33FREOtq7v0uBPWbQ" "\_IH6W4DoTRmdms0FJNlg4g" "iK\_3TbzcSj2-MbMXip\_XFg" "J4bjPFIcQ1ewrQqjN2qz6Q" "kfonMDBuR--UIhkyM2cWrg" "Kr6-9-3uR9Wp2923n-O2NA" "Nw\_9rjwvQ62u-HsuWIm53A" "QRmY8R2MQemuePb0EkYxWA" "usloSJzQRzCpOQ8bxKi2vA" "w9NGEWg-QiivMpjyurYKrA" "wKy-YzB-TK2lnK86Sx2RBA" "y2ZmJ-\_GRAmi3eHy1y8jzw" "ZmFj7w4hR5Cvy-owCLmZ1Q" "ZmlndPBLT-ivuOxm\_A7yDA" On Thursday, January 30, 2014 4:00:49 PM UTC-8, tallpsmith wrote: > If you can narrow down a specific few IDs of results that appear/disappear > based on the primary/replica shard, and confirm through an explicit GET of > that ID with the preference=\_local on the primary shard & replica for that > result. To work out which shard # a specific ID belongs to, you can run > this query: > > curl -XGET '_[http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1) > [http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1)_' -d ' > { > "fields" : , > "query" : { > "ids" : { > "values" : [ > "123456789" > ] > } > }, > "explain" : 1 > } > ' > > where the "values" attribute you place the ID of the item you're after. > Within the result response you'l see the shard Id, use that to identify > which host is the primary and which is the replica. You can then run the > GET query with the preference=\_local on each of those hosts and see if the > primary or replica shows the result. You will need to understand whether > the item that is 'flappy' (appearing/disappearing depending on the shard > being searched) is supposed to be in there or not, perhaps checking the > data store that is the source of the index (is it a dB?). > > We have very infrequent case where the replica shard is not properly > receiving a delete at least with 0.19.10. The delete successfully applies > to the Primary, but the Replica still holds the value and returns it within > search results. We have loads of insert/update/delete activity and the > number of flappy items is very small, but it is definitely a thing. > > see this previous thread: > [http://elasticsearch-users.115913.n3.nabble.com/Deleted-items-appears-when-searching-a-replica-shard-td4029075.html](http://elasticsearch-users.115913.n3.nabble.com/Deleted-items-appears-when-searching-a-replica-shard-td4029075.html) > > If it is the replica shard that's incorrect (my bet), the way to fix it is > to relocate the replica shard to another host. The relocation will take > the copy of the primary (correct copy) and recreate a new replica shard, > effectively neutralizing the inconsistency. > > We have written a tool, Scrutineer ([https://github.com/aconex/scrutineer](https://github.com/aconex/scrutineer)) > which can help detect inconsistencies in your cluster. I also have a tool > not yet published to github that can help check these Primary/Replica > inconsistencies if that would help (you pass a list of IDs to it and it'll > check whether they're flappy between the primary & replica or not). It can > also help automate the rebuilding of just the replica shards by shunting > them around (rather than a full rolling restart of ALL the shards, just the > shard replicas you want) > > cheers, > > Paul Smith > > On 31 January 2014 09:44, Binh Ly \<[bi...@hibalo.com](mailto:bi...@hibalo.com) \\> wrote: > > > Xavier, can you post an example of 1 full query and then also show how > > the results of this one query is inconsistent? Just trying to understand > > what is inconsistent. Thanks. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/038735ba-cef6-4634-9d46-7ff39dffc4d2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/038735ba-cef6-4634-9d46-7ff39dffc4d2%40googlegroups.com) > > . > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a99ab249-ddf4-4c38-97d7-3bfe8ec41b5f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a99ab249-ddf4-4c38-97d7-3bfe8ec41b5f%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [January 31, 2014, 12:59am UTC](https://discuss.elastic.co/t/primary-vs-replica-shard-inconsistencies/15514/5 "2014-01-31T00:59:04Z")
the flappy detection tool I have connects to the cluster using the standard java autodiscovery mechanism, and, works out which shards are involved, and then creates explicit TransportClient connection to each host, so would need access to 9300 (the SMILE based protocol port). Would that help? (is 9300 accessible from a host that can run java ? On 31 January 2014 11:45, [xavier@gaikai.com](mailto:xavier@gaikai.com) wrote: > We have 4 query heads total (esq1.r6, esq2.r6, esq3.r7, esq4.r7). > Interestingly query heads in the same rack give the same results. We don't > do deletes at all on these indices so that shouldn't be an issue. > Unfortunately at the moment I can't do preference=\_local while getting the > \_id(s) directly because we don't allow access on 9200 on our worker nodes. > I might be able to right some code to figure this out though. Either way > here's my id results from the different heads. > > esq2.r6 gets 28 total results > esq3.r7 gets 9 total results > > $curl -XGET " > [http://esq2.r6:9200/events/\_search?q=sessionId:1390953880&size=100](http://esq2.r6:9200/events/_search?q=sessionId:1390953880&size=100)" | jq > '.hits.hits.\_id' | sort > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 19337 100 19337 0 0 1039k 0 --:--:-- --:--:-- --:--:-- > 1049k > "0LcI\_px4SZy5ZQkI\_V7Qyw" > "1sAGREtMSfK8OIxZErm8RQ" > "6IV2v4TFTr-Gl1eC6hrj0Q" > "6nwMexTHQBmFxfykOgKqWA" > "7hFYs6y-QG6wGYEkoBKmdg" > "9MTM10SeQ2yqWIb08oPnFA" > "aELtGN6DQpmdRlQbr8i0uA" > "AUHUg6k0QZOf\_oGjsjSsGA" > "Bo\_u1eYGSF2LeU78kbcFZg" > "EWs1K8YsR9-IBSAWK6ld7A" > "Fx4l6\_axSGCxpyFm7C7BSQ" > "gpCrAZrNTNezWPfensER3g" > "HAFmGcWuQAylxGjmnZZkSQ" > "HB4Kwz3RSWWH5NHvyH4JMg" > "H-eP-33FREOtq7v0uBPWbQ" > "\_IH6W4DoTRmdms0FJNlg4g" > "iK\_3TbzcSj2-MbMXip\_XFg" > "J4bjPFIcQ1ewrQqjN2qz6Q" > "kfonMDBuR--UIhkyM2cWrg" > "Kr6-9-3uR9Wp2923n-O2NA" > "Nw\_9rjwvQ62u-HsuWIm53A" > "QRmY8R2MQemuePb0EkYxWA" > "usloSJzQRzCpOQ8bxKi2vA" > "w9NGEWg-QiivMpjyurYKrA" > "wKy-YzB-TK2lnK86Sx2RBA" > "y2ZmJ-\_GRAmi3eHy1y8jzw" > "ZmFj7w4hR5Cvy-owCLmZ1Q" > "ZmlndPBLT-ivuOxm\_A7yDA" > > $curl -XGET " > [http://esq3.r7:9200/events/\_search?q=sessionId:1390953880&size=100](http://esq3.r7:9200/events/_search?q=sessionId:1390953880&size=100)" | jq > '.hits.hits.\_id' | sort > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 6808 100 6808 0 0 70082 0 --:--:-- --:--:-- --:--:-- > 70185 > "1sAGREtMSfK8OIxZErm8RQ" > "7hFYs6y-QG6wGYEkoBKmdg" > "aELtGN6DQpmdRlQbr8i0uA" > "Fx4l6\_axSGCxpyFm7C7BSQ" > "HAFmGcWuQAylxGjmnZZkSQ" > "H-eP-33FREOtq7v0uBPWbQ" > "QRmY8R2MQemuePb0EkYxWA" > "wKy-YzB-TK2lnK86Sx2RBA" > "y2ZmJ-\_GRAmi3eHy1y8jzw" > > And here is es3.r7 with preference=\_primary\_first: > > $curl -XGET " > [http://esq3.r7/events/\_search?q=sessionId:1390953880&size=100&preference=\_primary\_first](http://esq3.r7/events/_search?q=sessionId:1390953880&size=100&preference=_primary_first)" > | jq '.hits.hits.\_id' | sort > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 100 19335 100 19335 0 0 871k 0 --:--:-- --:--:-- --:--:-- > 899k > "0LcI\_px4SZy5ZQkI\_V7Qyw" > "1sAGREtMSfK8OIxZErm8RQ" > "6IV2v4TFTr-Gl1eC6hrj0Q" > "6nwMexTHQBmFxfykOgKqWA" > "7hFYs6y-QG6wGYEkoBKmdg" > "9MTM10SeQ2yqWIb08oPnFA" > "aELtGN6DQpmdRlQbr8i0uA" > "AUHUg6k0QZOf\_oGjsjSsGA" > "Bo\_u1eYGSF2LeU78kbcFZg" > "EWs1K8YsR9-IBSAWK6ld7A" > "Fx4l6\_axSGCxpyFm7C7BSQ" > "gpCrAZrNTNezWPfensER3g" > "HAFmGcWuQAylxGjmnZZkSQ" > "HB4Kwz3RSWWH5NHvyH4JMg" > "H-eP-33FREOtq7v0uBPWbQ" > "\_IH6W4DoTRmdms0FJNlg4g" > "iK\_3TbzcSj2-MbMXip\_XFg" > "J4bjPFIcQ1ewrQqjN2qz6Q" > "kfonMDBuR--UIhkyM2cWrg" > "Kr6-9-3uR9Wp2923n-O2NA" > "Nw\_9rjwvQ62u-HsuWIm53A" > "QRmY8R2MQemuePb0EkYxWA" > "usloSJzQRzCpOQ8bxKi2vA" > "w9NGEWg-QiivMpjyurYKrA" > "wKy-YzB-TK2lnK86Sx2RBA" > "y2ZmJ-\_GRAmi3eHy1y8jzw" > "ZmFj7w4hR5Cvy-owCLmZ1Q" > "ZmlndPBLT-ivuOxm\_A7yDA" > > On Thursday, January 30, 2014 4:00:49 PM UTC-8, tallpsmith wrote: > > > If you can narrow down a specific few IDs of results that > > appear/disappear based on the primary/replica shard, and confirm through an > > explicit GET of that ID with the preference=\_local on the primary shard & > > replica for that result. To work out which shard # a specific ID belongs > > to, you can run this query: > > > > curl -XGET '_[http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1) > > [http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1)_' -d ' > > { > > "fields" : , > > "query" : { > > "ids" : { > > "values" : [ > > "123456789" > > ] > > } > > }, > > "explain" : 1 > > } > > ' > > > > where the "values" attribute you place the ID of the item you're after. > > Within the result response you'l see the shard Id, use that to identify > > which host is the primary and which is the replica. You can then run the > > GET query with the preference=\_local on each of those hosts and see if the > > primary or replica shows the result. You will need to understand whether > > the item that is 'flappy' (appearing/disappearing depending on the shard > > being searched) is supposed to be in there or not, perhaps checking the > > data store that is the source of the index (is it a dB?). > > > > We have very infrequent case where the replica shard is not properly > > receiving a delete at least with 0.19.10. The delete successfully applies > > to the Primary, but the Replica still holds the value and returns it within > > search results. We have loads of insert/update/delete activity and the > > number of flappy items is very small, but it is definitely a thing. > > > > see this previous thread: [http://elasticsearch-users](http://elasticsearch-users). > > [115913.n3.nabble.com/Deleted-items-appears-when-searching-](http://115913.n3.nabble.com/Deleted-items-appears-when-searching-) > > a-replica-shard-td4029075.html > > > > If it is the replica shard that's incorrect (my bet), the way to fix it > > is to relocate the replica shard to another host. The relocation will take > > the copy of the primary (correct copy) and recreate a new replica shard, > > effectively neutralizing the inconsistency. > > > > We have written a tool, Scrutineer ([https://github.com/aconex/scrutineer](https://github.com/aconex/scrutineer)) > > which can help detect inconsistencies in your cluster. I also have a tool > > not yet published to github that can help check these Primary/Replica > > inconsistencies if that would help (you pass a list of IDs to it and it'll > > check whether they're flappy between the primary & replica or not). It can > > also help automate the rebuilding of just the replica shards by shunting > > them around (rather than a full rolling restart of ALL the shards, just the > > shard replicas you want) > > > > cheers, > > > > Paul Smith > > > > On 31 January 2014 09:44, Binh Ly [bi...@hibalo.com](mailto:bi...@hibalo.com) wrote: > > > > > Xavier, can you post an example of 1 full query and then also show how > > > the results of this one query is inconsistent? Just trying to understand > > > what is inconsistent. Thanks. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/038735ba-cef6-4634-9d46-7ff39dffc4d2% > > > [40googlegroups.com](http://40googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/a99ab249-ddf4-4c38-97d7-3bfe8ec41b5f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a99ab249-ddf4-4c38-97d7-3bfe8ec41b5f%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHfYWB50LGse5BghNJjo8a-RVkYtRiyVpS-p5d%3DH%3DZGWxk7PAg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHfYWB50LGse5BghNJjo8a-RVkYtRiyVpS-p5d%3DH%3DZGWxk7PAg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![xavier1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xavier1/32/1836_2.png) [@xavier1](https://discuss.elastic.co/u/xavier1)\ **Post date:** [January 31, 2014, 1:28am UTC](https://discuss.elastic.co/t/primary-vs-replica-shard-inconsistencies/15514/6 "2014-01-31T01:28:42Z")
Yeah, that should work. I'll take a look at that and see if it can help pinpoint problematic shards. On Thursday, January 30, 2014 4:59:04 PM UTC-8, tallpsmith wrote: > the flappy detection tool I have connects to the cluster using the > standard java autodiscovery mechanism, and, works out which shards are > involved, and then creates explicit TransportClient connection to each > host, so would need access to 9300 (the SMILE based protocol port). Would > that help? (is 9300 accessible from a host that can run java ? > > On 31 January 2014 11:45, \<[xav...@gaikai.com](mailto:xav...@gaikai.com) \\> wrote: > > > We have 4 query heads total (esq1.r6, esq2.r6, esq3.r7, esq4.r7). > > Interestingly query heads in the same rack give the same results. We don't > > do deletes at all on these indices so that shouldn't be an issue. > > Unfortunately at the moment I can't do preference=\_local while getting the > > \_id(s) directly because we don't allow access on 9200 on our worker nodes. > > I might be able to right some code to figure this out though. Either way > > here's my id results from the different heads. > > > > esq2.r6 gets 28 total results > > esq3.r7 gets 9 total results > > > > $curl -XGET " > > [http://esq2.r6:9200/events/\_search?q=sessionId:1390953880&size=100](http://esq2.r6:9200/events/_search?q=sessionId:1390953880&size=100)" | jq > > '.hits.hits.\_id' | sort > > % Total % Received % Xferd Average Speed Time Time Time > > Current > > Dload Upload Total Spent Left > > Speed > > 100 19337 100 19337 0 0 1039k 0 --:--:-- --:--:-- --:--:-- > > 1049k > > "0LcI\_px4SZy5ZQkI\_V7Qyw" > > "1sAGREtMSfK8OIxZErm8RQ" > > "6IV2v4TFTr-Gl1eC6hrj0Q" > > "6nwMexTHQBmFxfykOgKqWA" > > "7hFYs6y-QG6wGYEkoBKmdg" > > "9MTM10SeQ2yqWIb08oPnFA" > > "aELtGN6DQpmdRlQbr8i0uA" > > "AUHUg6k0QZOf\_oGjsjSsGA" > > "Bo\_u1eYGSF2LeU78kbcFZg" > > "EWs1K8YsR9-IBSAWK6ld7A" > > "Fx4l6\_axSGCxpyFm7C7BSQ" > > "gpCrAZrNTNezWPfensER3g" > > "HAFmGcWuQAylxGjmnZZkSQ" > > "HB4Kwz3RSWWH5NHvyH4JMg" > > "H-eP-33FREOtq7v0uBPWbQ" > > "\_IH6W4DoTRmdms0FJNlg4g" > > "iK\_3TbzcSj2-MbMXip\_XFg" > > "J4bjPFIcQ1ewrQqjN2qz6Q" > > "kfonMDBuR--UIhkyM2cWrg" > > "Kr6-9-3uR9Wp2923n-O2NA" > > "Nw\_9rjwvQ62u-HsuWIm53A" > > "QRmY8R2MQemuePb0EkYxWA" > > "usloSJzQRzCpOQ8bxKi2vA" > > "w9NGEWg-QiivMpjyurYKrA" > > "wKy-YzB-TK2lnK86Sx2RBA" > > "y2ZmJ-\_GRAmi3eHy1y8jzw" > > "ZmFj7w4hR5Cvy-owCLmZ1Q" > > "ZmlndPBLT-ivuOxm\_A7yDA" > > > > $curl -XGET " > > [http://esq3.r7:9200/events/\_search?q=sessionId:1390953880&size=100](http://esq3.r7:9200/events/_search?q=sessionId:1390953880&size=100)" | jq > > '.hits.hits.\_id' | sort > > % Total % Received % Xferd Average Speed Time Time Time > > Current > > Dload Upload Total Spent Left > > Speed > > 100 6808 100 6808 0 0 70082 0 --:--:-- --:--:-- --:--:-- > > 70185 > > "1sAGREtMSfK8OIxZErm8RQ" > > "7hFYs6y-QG6wGYEkoBKmdg" > > "aELtGN6DQpmdRlQbr8i0uA" > > "Fx4l6\_axSGCxpyFm7C7BSQ" > > "HAFmGcWuQAylxGjmnZZkSQ" > > "H-eP-33FREOtq7v0uBPWbQ" > > "QRmY8R2MQemuePb0EkYxWA" > > "wKy-YzB-TK2lnK86Sx2RBA" > > "y2ZmJ-\_GRAmi3eHy1y8jzw" > > > > And here is es3.r7 with preference=\_primary\_first: > > > > $curl -XGET " > > [http://esq3.r7/events/\_search?q=sessionId:1390953880&size=100&preference=\_primary\_first](http://esq3.r7/events/_search?q=sessionId:1390953880&size=100&preference=_primary_first)" > > | jq '.hits.hits.\_id' | sort > > % Total % Received % Xferd Average Speed Time Time Time > > Current > > Dload Upload Total Spent Left > > Speed > > 100 19335 100 19335 0 0 871k 0 --:--:-- --:--:-- --:--:-- > > 899k > > "0LcI\_px4SZy5ZQkI\_V7Qyw" > > "1sAGREtMSfK8OIxZErm8RQ" > > "6IV2v4TFTr-Gl1eC6hrj0Q" > > "6nwMexTHQBmFxfykOgKqWA" > > "7hFYs6y-QG6wGYEkoBKmdg" > > "9MTM10SeQ2yqWIb08oPnFA" > > "aELtGN6DQpmdRlQbr8i0uA" > > "AUHUg6k0QZOf\_oGjsjSsGA" > > "Bo\_u1eYGSF2LeU78kbcFZg" > > "EWs1K8YsR9-IBSAWK6ld7A" > > "Fx4l6\_axSGCxpyFm7C7BSQ" > > "gpCrAZrNTNezWPfensER3g" > > "HAFmGcWuQAylxGjmnZZkSQ" > > "HB4Kwz3RSWWH5NHvyH4JMg" > > "H-eP-33FREOtq7v0uBPWbQ" > > "\_IH6W4DoTRmdms0FJNlg4g" > > "iK\_3TbzcSj2-MbMXip\_XFg" > > "J4bjPFIcQ1ewrQqjN2qz6Q" > > "kfonMDBuR--UIhkyM2cWrg" > > "Kr6-9-3uR9Wp2923n-O2NA" > > "Nw\_9rjwvQ62u-HsuWIm53A" > > "QRmY8R2MQemuePb0EkYxWA" > > "usloSJzQRzCpOQ8bxKi2vA" > > "w9NGEWg-QiivMpjyurYKrA" > > "wKy-YzB-TK2lnK86Sx2RBA" > > "y2ZmJ-\_GRAmi3eHy1y8jzw" > > "ZmFj7w4hR5Cvy-owCLmZ1Q" > > "ZmlndPBLT-ivuOxm\_A7yDA" > > > > On Thursday, January 30, 2014 4:00:49 PM UTC-8, tallpsmith wrote: > > > > > If you can narrow down a specific few IDs of results that > > > appear/disappear based on the primary/replica shard, and confirm through an > > > explicit GET of that ID with the preference=\_local on the primary shard & > > > replica for that result. To work out which shard # a specific ID belongs > > > to, you can run this query: > > > > > > curl -XGET '_[http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1) > > > [http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1)_' -d ' > > > { > > > "fields" : , > > > "query" : { > > > "ids" : { > > > "values" : [ > > > "123456789" > > > ] > > > } > > > }, > > > "explain" : 1 > > > } > > > ' > > > > > > where the "values" attribute you place the ID of the item you're after. > > > Within the result response you'l see the shard Id, use that to identify > > > which host is the primary and which is the replica. You can then run the > > > GET query with the preference=\_local on each of those hosts and see if the > > > primary or replica shows the result. You will need to understand whether > > > the item that is 'flappy' (appearing/disappearing depending on the shard > > > being searched) is supposed to be in there or not, perhaps checking the > > > data store that is the source of the index (is it a dB?). > > > > > > We have very infrequent case where the replica shard is not properly > > > receiving a delete at least with 0.19.10. The delete successfully applies > > > to the Primary, but the Replica still holds the value and returns it within > > > search results. We have loads of insert/update/delete activity and the > > > number of flappy items is very small, but it is definitely a thing. > > > > > > see this previous thread: [http://elasticsearch-users](http://elasticsearch-users). > > > [115913.n3.nabble.com/Deleted-items-appears-when-searching-](http://115913.n3.nabble.com/Deleted-items-appears-when-searching-) > > > a-replica-shard-td4029075.html > > > > > > If it is the replica shard that's incorrect (my bet), the way to fix it > > > is to relocate the replica shard to another host. The relocation will take > > > the copy of the primary (correct copy) and recreate a new replica shard, > > > effectively neutralizing the inconsistency. > > > > > > We have written a tool, Scrutineer ([https://github.com/aconex/scrutineer](https://github.com/aconex/scrutineer)) > > > which can help detect inconsistencies in your cluster. I also have a tool > > > not yet published to github that can help check these Primary/Replica > > > inconsistencies if that would help (you pass a list of IDs to it and it'll > > > check whether they're flappy between the primary & replica or not). It can > > > also help automate the rebuilding of just the replica shards by shunting > > > them around (rather than a full rolling restart of ALL the shards, just the > > > shard replicas you want) > > > > > > cheers, > > > > > > Paul Smith > > > > > > On 31 January 2014 09:44, Binh Ly [bi...@hibalo.com](mailto:bi...@hibalo.com) wrote: > > > > > > > Xavier, can you post an example of 1 full query and then also show how > > > > the results of this one query is inconsistent? Just trying to understand > > > > what is inconsistent. Thanks. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/038735ba-cef6-4634-9d46-7ff39dffc4d2% > > > > [40googlegroups.com](http://40googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/a99ab249-ddf4-4c38-97d7-3bfe8ec41b5f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a99ab249-ddf4-4c38-97d7-3bfe8ec41b5f%40googlegroups.com) > > > . > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a61c780e-b26b-458d-8d0c-c7e19f1bab18%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a61c780e-b26b-458d-8d0c-c7e19f1bab18%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [January 31, 2014, 1:29am UTC](https://discuss.elastic.co/t/primary-vs-replica-shard-inconsistencies/15514/7 "2014-01-31T01:29:17Z")
if it helps at all, i've pushed the flappy item detector tool (_cough_) here: [https://github.com/Aconex/es-flappyitem-detector](https://github.com/Aconex/es-flappyitem-detector) We have a simple 3-node cluster, 5 shards, 1 replica, so I'm sure there's code in there that is built around those assumptions, but should be easily modified to suit your purpose perhaps. cheers, Paul On 31 January 2014 11:59, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote: > the flappy detection tool I have connects to the cluster using the > standard java autodiscovery mechanism, and, works out which shards are > involved, and then creates explicit TransportClient connection to each > host, so would need access to 9300 (the SMILE based protocol port). Would > that help? (is 9300 accessible from a host that can run java ? > > On 31 January 2014 11:45, [xavier@gaikai.com](mailto:xavier@gaikai.com) wrote: > > > We have 4 query heads total (esq1.r6, esq2.r6, esq3.r7, esq4.r7). > > Interestingly query heads in the same rack give the same results. We don't > > do deletes at all on these indices so that shouldn't be an issue. > > Unfortunately at the moment I can't do preference=\_local while getting the > > \_id(s) directly because we don't allow access on 9200 on our worker nodes. > > I might be able to right some code to figure this out though. Either way > > here's my id results from the different heads. > > > > esq2.r6 gets 28 total results > > esq3.r7 gets 9 total results > > > > $curl -XGET " > > [http://esq2.r6:9200/events/\_search?q=sessionId:1390953880&size=100](http://esq2.r6:9200/events/_search?q=sessionId:1390953880&size=100)" | jq > > '.hits.hits.\_id' | sort > > % Total % Received % Xferd Average Speed Time Time Time > > Current > > Dload Upload Total Spent Left > > Speed > > 100 19337 100 19337 0 0 1039k 0 --:--:-- --:--:-- --:--:-- > > 1049k > > "0LcI\_px4SZy5ZQkI\_V7Qyw" > > "1sAGREtMSfK8OIxZErm8RQ" > > "6IV2v4TFTr-Gl1eC6hrj0Q" > > "6nwMexTHQBmFxfykOgKqWA" > > "7hFYs6y-QG6wGYEkoBKmdg" > > "9MTM10SeQ2yqWIb08oPnFA" > > "aELtGN6DQpmdRlQbr8i0uA" > > "AUHUg6k0QZOf\_oGjsjSsGA" > > "Bo\_u1eYGSF2LeU78kbcFZg" > > "EWs1K8YsR9-IBSAWK6ld7A" > > "Fx4l6\_axSGCxpyFm7C7BSQ" > > "gpCrAZrNTNezWPfensER3g" > > "HAFmGcWuQAylxGjmnZZkSQ" > > "HB4Kwz3RSWWH5NHvyH4JMg" > > "H-eP-33FREOtq7v0uBPWbQ" > > "\_IH6W4DoTRmdms0FJNlg4g" > > "iK\_3TbzcSj2-MbMXip\_XFg" > > "J4bjPFIcQ1ewrQqjN2qz6Q" > > "kfonMDBuR--UIhkyM2cWrg" > > "Kr6-9-3uR9Wp2923n-O2NA" > > "Nw\_9rjwvQ62u-HsuWIm53A" > > "QRmY8R2MQemuePb0EkYxWA" > > "usloSJzQRzCpOQ8bxKi2vA" > > "w9NGEWg-QiivMpjyurYKrA" > > "wKy-YzB-TK2lnK86Sx2RBA" > > "y2ZmJ-\_GRAmi3eHy1y8jzw" > > "ZmFj7w4hR5Cvy-owCLmZ1Q" > > "ZmlndPBLT-ivuOxm\_A7yDA" > > > > $curl -XGET " > > [http://esq3.r7:9200/events/\_search?q=sessionId:1390953880&size=100](http://esq3.r7:9200/events/_search?q=sessionId:1390953880&size=100)" | jq > > '.hits.hits.\_id' | sort > > % Total % Received % Xferd Average Speed Time Time Time > > Current > > Dload Upload Total Spent Left > > Speed > > 100 6808 100 6808 0 0 70082 0 --:--:-- --:--:-- --:--:-- > > 70185 > > "1sAGREtMSfK8OIxZErm8RQ" > > "7hFYs6y-QG6wGYEkoBKmdg" > > "aELtGN6DQpmdRlQbr8i0uA" > > "Fx4l6\_axSGCxpyFm7C7BSQ" > > "HAFmGcWuQAylxGjmnZZkSQ" > > "H-eP-33FREOtq7v0uBPWbQ" > > "QRmY8R2MQemuePb0EkYxWA" > > "wKy-YzB-TK2lnK86Sx2RBA" > > "y2ZmJ-\_GRAmi3eHy1y8jzw" > > > > And here is es3.r7 with preference=\_primary\_first: > > > > $curl -XGET " > > [http://esq3.r7/events/\_search?q=sessionId:1390953880&size=100&preference=\_primary\_first](http://esq3.r7/events/_search?q=sessionId:1390953880&size=100&preference=_primary_first)" > > | jq '.hits.hits.\_id' | sort > > % Total % Received % Xferd Average Speed Time Time Time > > Current > > Dload Upload Total Spent Left > > Speed > > 100 19335 100 19335 0 0 871k 0 --:--:-- --:--:-- --:--:-- > > 899k > > "0LcI\_px4SZy5ZQkI\_V7Qyw" > > "1sAGREtMSfK8OIxZErm8RQ" > > "6IV2v4TFTr-Gl1eC6hrj0Q" > > "6nwMexTHQBmFxfykOgKqWA" > > "7hFYs6y-QG6wGYEkoBKmdg" > > "9MTM10SeQ2yqWIb08oPnFA" > > "aELtGN6DQpmdRlQbr8i0uA" > > "AUHUg6k0QZOf\_oGjsjSsGA" > > "Bo\_u1eYGSF2LeU78kbcFZg" > > "EWs1K8YsR9-IBSAWK6ld7A" > > "Fx4l6\_axSGCxpyFm7C7BSQ" > > "gpCrAZrNTNezWPfensER3g" > > "HAFmGcWuQAylxGjmnZZkSQ" > > "HB4Kwz3RSWWH5NHvyH4JMg" > > "H-eP-33FREOtq7v0uBPWbQ" > > "\_IH6W4DoTRmdms0FJNlg4g" > > "iK\_3TbzcSj2-MbMXip\_XFg" > > "J4bjPFIcQ1ewrQqjN2qz6Q" > > "kfonMDBuR--UIhkyM2cWrg" > > "Kr6-9-3uR9Wp2923n-O2NA" > > "Nw\_9rjwvQ62u-HsuWIm53A" > > "QRmY8R2MQemuePb0EkYxWA" > > "usloSJzQRzCpOQ8bxKi2vA" > > "w9NGEWg-QiivMpjyurYKrA" > > "wKy-YzB-TK2lnK86Sx2RBA" > > "y2ZmJ-\_GRAmi3eHy1y8jzw" > > "ZmFj7w4hR5Cvy-owCLmZ1Q" > > "ZmlndPBLT-ivuOxm\_A7yDA" > > > > On Thursday, January 30, 2014 4:00:49 PM UTC-8, tallpsmith wrote: > > > > > If you can narrow down a specific few IDs of results that > > > appear/disappear based on the primary/replica shard, and confirm through an > > > explicit GET of that ID with the preference=\_local on the primary shard & > > > replica for that result. To work out which shard # a specific ID belongs > > > to, you can run this query: > > > > > > curl -XGET '_[http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1) > > > [http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1)_' -d ' > > > { > > > "fields" : , > > > "query" : { > > > "ids" : { > > > "values" : [ > > > "123456789" > > > ] > > > } > > > }, > > > "explain" : 1 > > > } > > > ' > > > > > > where the "values" attribute you place the ID of the item you're after. > > > Within the result response you'l see the shard Id, use that to identify > > > which host is the primary and which is the replica. You can then run the > > > GET query with the preference=\_local on each of those hosts and see if the > > > primary or replica shows the result. You will need to understand whether > > > the item that is 'flappy' (appearing/disappearing depending on the shard > > > being searched) is supposed to be in there or not, perhaps checking the > > > data store that is the source of the index (is it a dB?). > > > > > > We have very infrequent case where the replica shard is not properly > > > receiving a delete at least with 0.19.10. The delete successfully applies > > > to the Primary, but the Replica still holds the value and returns it within > > > search results. We have loads of insert/update/delete activity and the > > > number of flappy items is very small, but it is definitely a thing. > > > > > > see this previous thread: [http://elasticsearch-users](http://elasticsearch-users). > > > [115913.n3.nabble.com/Deleted-items-appears-when-searching-](http://115913.n3.nabble.com/Deleted-items-appears-when-searching-) > > > a-replica-shard-td4029075.html > > > > > > If it is the replica shard that's incorrect (my bet), the way to fix it > > > is to relocate the replica shard to another host. The relocation will take > > > the copy of the primary (correct copy) and recreate a new replica shard, > > > effectively neutralizing the inconsistency. > > > > > > We have written a tool, Scrutineer ([https://github.com/aconex/scrutineer](https://github.com/aconex/scrutineer)) > > > which can help detect inconsistencies in your cluster. I also have a tool > > > not yet published to github that can help check these Primary/Replica > > > inconsistencies if that would help (you pass a list of IDs to it and it'll > > > check whether they're flappy between the primary & replica or not). It can > > > also help automate the rebuilding of just the replica shards by shunting > > > them around (rather than a full rolling restart of ALL the shards, just the > > > shard replicas you want) > > > > > > cheers, > > > > > > Paul Smith > > > > > > On 31 January 2014 09:44, Binh Ly [bi...@hibalo.com](mailto:bi...@hibalo.com) wrote: > > > > > > > Xavier, can you post an example of 1 full query and then also show how > > > > the results of this one query is inconsistent? Just trying to understand > > > > what is inconsistent. Thanks. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/038735ba-cef6-4634-9d46-7ff39dffc4d2% > > > > [40googlegroups.com](http://40googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/a99ab249-ddf4-4c38-97d7-3bfe8ec41b5f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a99ab249-ddf4-4c38-97d7-3bfe8ec41b5f%40googlegroups.com) > > > . > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHfYWB5%3DUVzDmvmNHHmhkGnFaEAV37-c2cnMPKSvuqsJ3MzC2Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHfYWB5%3DUVzDmvmNHHmhkGnFaEAV37-c2cnMPKSvuqsJ3MzC2Q%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [January 31, 2014, 1:38am UTC](https://discuss.elastic.co/t/primary-vs-replica-shard-inconsistencies/15514/8 "2014-01-31T01:38:11Z")
if you do use it, don't forget we build for ES 0.19, so change the pom.xml to your ES version otherwise it won't connect... ๐Ÿ™‚ On 31 January 2014 12:29, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote: > if it helps at all, i've pushed the flappy item detector tool (_cough_) > here: > > [https://github.com/Aconex/es-flappyitem-detector](https://github.com/Aconex/es-flappyitem-detector) > > We have a simple 3-node cluster, 5 shards, 1 replica, so I'm sure there's > code in there that is built around those assumptions, but should be easily > modified to suit your purpose perhaps. > > cheers, > > Paul > > On 31 January 2014 11:59, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote: > > > the flappy detection tool I have connects to the cluster using the > > standard java autodiscovery mechanism, and, works out which shards are > > involved, and then creates explicit TransportClient connection to each > > host, so would need access to 9300 (the SMILE based protocol port). Would > > that help? (is 9300 accessible from a host that can run java ? > > > > On 31 January 2014 11:45, [xavier@gaikai.com](mailto:xavier@gaikai.com) wrote: > > > > > We have 4 query heads total (esq1.r6, esq2.r6, esq3.r7, esq4.r7). > > > Interestingly query heads in the same rack give the same results. We don't > > > do deletes at all on these indices so that shouldn't be an issue. > > > Unfortunately at the moment I can't do preference=\_local while getting the > > > \_id(s) directly because we don't allow access on 9200 on our worker nodes. > > > I might be able to right some code to figure this out though. Either way > > > here's my id results from the different heads. > > > > > > esq2.r6 gets 28 total results > > > esq3.r7 gets 9 total results > > > > > > $curl -XGET " > > > [http://esq2.r6:9200/events/\_search?q=sessionId:1390953880&size=100](http://esq2.r6:9200/events/_search?q=sessionId:1390953880&size=100)" | > > > jq '.hits.hits.\_id' | sort > > > % Total % Received % Xferd Average Speed Time Time Time > > > Current > > > Dload Upload Total Spent Left > > > Speed > > > 100 19337 100 19337 0 0 1039k 0 --:--:-- --:--:-- --:--:-- > > > 1049k > > > "0LcI\_px4SZy5ZQkI\_V7Qyw" > > > "1sAGREtMSfK8OIxZErm8RQ" > > > "6IV2v4TFTr-Gl1eC6hrj0Q" > > > "6nwMexTHQBmFxfykOgKqWA" > > > "7hFYs6y-QG6wGYEkoBKmdg" > > > "9MTM10SeQ2yqWIb08oPnFA" > > > "aELtGN6DQpmdRlQbr8i0uA" > > > "AUHUg6k0QZOf\_oGjsjSsGA" > > > "Bo\_u1eYGSF2LeU78kbcFZg" > > > "EWs1K8YsR9-IBSAWK6ld7A" > > > "Fx4l6\_axSGCxpyFm7C7BSQ" > > > "gpCrAZrNTNezWPfensER3g" > > > "HAFmGcWuQAylxGjmnZZkSQ" > > > "HB4Kwz3RSWWH5NHvyH4JMg" > > > "H-eP-33FREOtq7v0uBPWbQ" > > > "\_IH6W4DoTRmdms0FJNlg4g" > > > "iK\_3TbzcSj2-MbMXip\_XFg" > > > "J4bjPFIcQ1ewrQqjN2qz6Q" > > > "kfonMDBuR--UIhkyM2cWrg" > > > "Kr6-9-3uR9Wp2923n-O2NA" > > > "Nw\_9rjwvQ62u-HsuWIm53A" > > > "QRmY8R2MQemuePb0EkYxWA" > > > "usloSJzQRzCpOQ8bxKi2vA" > > > "w9NGEWg-QiivMpjyurYKrA" > > > "wKy-YzB-TK2lnK86Sx2RBA" > > > "y2ZmJ-\_GRAmi3eHy1y8jzw" > > > "ZmFj7w4hR5Cvy-owCLmZ1Q" > > > "ZmlndPBLT-ivuOxm\_A7yDA" > > > > > > $curl -XGET " > > > [http://esq3.r7:9200/events/\_search?q=sessionId:1390953880&size=100](http://esq3.r7:9200/events/_search?q=sessionId:1390953880&size=100)" | > > > jq '.hits.hits.\_id' | sort > > > % Total % Received % Xferd Average Speed Time Time Time > > > Current > > > Dload Upload Total Spent Left > > > Speed > > > 100 6808 100 6808 0 0 70082 0 --:--:-- --:--:-- --:--:-- > > > 70185 > > > "1sAGREtMSfK8OIxZErm8RQ" > > > "7hFYs6y-QG6wGYEkoBKmdg" > > > "aELtGN6DQpmdRlQbr8i0uA" > > > "Fx4l6\_axSGCxpyFm7C7BSQ" > > > "HAFmGcWuQAylxGjmnZZkSQ" > > > "H-eP-33FREOtq7v0uBPWbQ" > > > "QRmY8R2MQemuePb0EkYxWA" > > > "wKy-YzB-TK2lnK86Sx2RBA" > > > "y2ZmJ-\_GRAmi3eHy1y8jzw" > > > > > > And here is es3.r7 with preference=\_primary\_first: > > > > > > $curl -XGET " > > > [http://esq3.r7/events/\_search?q=sessionId:1390953880&size=100&preference=\_primary\_first](http://esq3.r7/events/_search?q=sessionId:1390953880&size=100&preference=_primary_first)" > > > | jq '.hits.hits.\_id' | sort > > > % Total % Received % Xferd Average Speed Time Time Time > > > Current > > > Dload Upload Total Spent Left > > > Speed > > > 100 19335 100 19335 0 0 871k 0 --:--:-- --:--:-- --:--:-- > > > 899k > > > "0LcI\_px4SZy5ZQkI\_V7Qyw" > > > "1sAGREtMSfK8OIxZErm8RQ" > > > "6IV2v4TFTr-Gl1eC6hrj0Q" > > > "6nwMexTHQBmFxfykOgKqWA" > > > "7hFYs6y-QG6wGYEkoBKmdg" > > > "9MTM10SeQ2yqWIb08oPnFA" > > > "aELtGN6DQpmdRlQbr8i0uA" > > > "AUHUg6k0QZOf\_oGjsjSsGA" > > > "Bo\_u1eYGSF2LeU78kbcFZg" > > > "EWs1K8YsR9-IBSAWK6ld7A" > > > "Fx4l6\_axSGCxpyFm7C7BSQ" > > > "gpCrAZrNTNezWPfensER3g" > > > "HAFmGcWuQAylxGjmnZZkSQ" > > > "HB4Kwz3RSWWH5NHvyH4JMg" > > > "H-eP-33FREOtq7v0uBPWbQ" > > > "\_IH6W4DoTRmdms0FJNlg4g" > > > "iK\_3TbzcSj2-MbMXip\_XFg" > > > "J4bjPFIcQ1ewrQqjN2qz6Q" > > > "kfonMDBuR--UIhkyM2cWrg" > > > "Kr6-9-3uR9Wp2923n-O2NA" > > > "Nw\_9rjwvQ62u-HsuWIm53A" > > > "QRmY8R2MQemuePb0EkYxWA" > > > "usloSJzQRzCpOQ8bxKi2vA" > > > "w9NGEWg-QiivMpjyurYKrA" > > > "wKy-YzB-TK2lnK86Sx2RBA" > > > "y2ZmJ-\_GRAmi3eHy1y8jzw" > > > "ZmFj7w4hR5Cvy-owCLmZ1Q" > > > "ZmlndPBLT-ivuOxm\_A7yDA" > > > > > > On Thursday, January 30, 2014 4:00:49 PM UTC-8, tallpsmith wrote: > > > > > > > If you can narrow down a specific few IDs of results that > > > > appear/disappear based on the primary/replica shard, and confirm through an > > > > explicit GET of that ID with the preference=\_local on the primary shard & > > > > replica for that result. To work out which shard # a specific ID belongs > > > > to, you can run this query: > > > > > > > > curl -XGET '_[http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1) > > > > [http://127.0.0.1:9200/\_all/\_search?pretty=1](http://127.0.0.1:9200/_all/_search?pretty=1)_' -d ' > > > > { > > > > "fields" : , > > > > "query" : { > > > > "ids" : { > > > > "values" : [ > > > > "123456789" > > > > ] > > > > } > > > > }, > > > > "explain" : 1 > > > > } > > > > ' > > > > > > > > where the "values" attribute you place the ID of the item you're after. > > > > Within the result response you'l see the shard Id, use that to identify > > > > which host is the primary and which is the replica. You can then run the > > > > GET query with the preference=\_local on each of those hosts and see if the > > > > primary or replica shows the result. You will need to understand whether > > > > the item that is 'flappy' (appearing/disappearing depending on the shard > > > > being searched) is supposed to be in there or not, perhaps checking the > > > > data store that is the source of the index (is it a dB?). > > > > > > > > We have very infrequent case where the replica shard is not properly > > > > receiving a delete at least with 0.19.10. The delete successfully applies > > > > to the Primary, but the Replica still holds the value and returns it within > > > > search results. We have loads of insert/update/delete activity and the > > > > number of flappy items is very small, but it is definitely a thing. > > > > > > > > see this previous thread: [http://elasticsearch-users](http://elasticsearch-users). > > > > [115913.n3.nabble.com/Deleted-items-appears-when-searching-](http://115913.n3.nabble.com/Deleted-items-appears-when-searching-) > > > > a-replica-shard-td4029075.html > > > > > > > > If it is the replica shard that's incorrect (my bet), the way to fix it > > > > is to relocate the replica shard to another host. The relocation will take > > > > the copy of the primary (correct copy) and recreate a new replica shard, > > > > effectively neutralizing the inconsistency. > > > > > > > > We have written a tool, Scrutineer ([Aconex ยท GitHub](https://github.com/aconex/) > > > > scrutineer) which can help detect inconsistencies in your cluster. I > > > > also have a tool not yet published to github that can help check these > > > > Primary/Replica inconsistencies if that would help (you pass a list of IDs > > > > to it and it'll check whether they're flappy between the primary & replica > > > > or not). It can also help automate the rebuilding of just the replica > > > > shards by shunting them around (rather than a full rolling restart of ALL > > > > the shards, just the shard replicas you want) > > > > > > > > cheers, > > > > > > > > Paul Smith > > > > > > > > On 31 January 2014 09:44, Binh Ly [bi...@hibalo.com](mailto:bi...@hibalo.com) wrote: > > > > > > > > > Xavier, can you post an example of 1 full query and then also show how > > > > > the results of this one query is inconsistent? Just trying to understand > > > > > what is inconsistent. Thanks. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/038735ba-cef6-4634-9d46-7ff39dffc4d2% > > > > > [40googlegroups.com](http://40googlegroups.com). > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/a99ab249-ddf4-4c38-97d7-3bfe8ec41b5f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a99ab249-ddf4-4c38-97d7-3bfe8ec41b5f%40googlegroups.com) > > > > . > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHfYWB7J2%2B%2B5c6SC2AUm4jDiukLsmm8V7KN35PycSBOBMVsCoA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHfYWB7J2%2B%2B5c6SC2AUm4jDiukLsmm8V7KN35PycSBOBMVsCoA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:53am UTC](https://discuss.elastic.co/t/primary-vs-replica-shard-inconsistencies/15514/9 "2017-07-06T01:53:46Z")