We have 4 query heads total (esq1.r6, esq2.r6, esq3.r7, esq4.r7).
Interestingly query heads in the same rack give the same results. We don't
do deletes at all on these indices so that shouldn't be an issue.
Unfortunately at the moment I can't do preference=_local while getting the
_id(s) directly because we don't allow access on 9200 on our worker nodes.
I might be able to right some code to figure this out though. Either way
here's my id results from the different heads.
esq2.r6 gets 28 total results
esq3.r7 gets 9 total results
$curl -XGET
"http://esq2.r6:9200/events/_search?q=sessionId:1390953880&size=100" | jq
'.hits.hits._id' | sort
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 19337 100 19337 0 0 1039k 0 --:--:-- --:--:-- --:--:--
1049k
"0LcI_px4SZy5ZQkI_V7Qyw"
"1sAGREtMSfK8OIxZErm8RQ"
"6IV2v4TFTr-Gl1eC6hrj0Q"
"6nwMexTHQBmFxfykOgKqWA"
"7hFYs6y-QG6wGYEkoBKmdg"
"9MTM10SeQ2yqWIb08oPnFA"
"aELtGN6DQpmdRlQbr8i0uA"
"AUHUg6k0QZOf_oGjsjSsGA"
"Bo_u1eYGSF2LeU78kbcFZg"
"EWs1K8YsR9-IBSAWK6ld7A"
"Fx4l6_axSGCxpyFm7C7BSQ"
"gpCrAZrNTNezWPfensER3g"
"HAFmGcWuQAylxGjmnZZkSQ"
"HB4Kwz3RSWWH5NHvyH4JMg"
"H-eP-33FREOtq7v0uBPWbQ"
"_IH6W4DoTRmdms0FJNlg4g"
"iK_3TbzcSj2-MbMXip_XFg"
"J4bjPFIcQ1ewrQqjN2qz6Q"
"kfonMDBuR--UIhkyM2cWrg"
"Kr6-9-3uR9Wp2923n-O2NA"
"Nw_9rjwvQ62u-HsuWIm53A"
"QRmY8R2MQemuePb0EkYxWA"
"usloSJzQRzCpOQ8bxKi2vA"
"w9NGEWg-QiivMpjyurYKrA"
"wKy-YzB-TK2lnK86Sx2RBA"
"y2ZmJ-_GRAmi3eHy1y8jzw"
"ZmFj7w4hR5Cvy-owCLmZ1Q"
"ZmlndPBLT-ivuOxm_A7yDA"
$curl -XGET
"http://esq3.r7:9200/events/_search?q=sessionId:1390953880&size=100" | jq
'.hits.hits._id' | sort
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 6808 100 6808 0 0 70082 0 --:--:-- --:--:-- --:--:--
70185
"1sAGREtMSfK8OIxZErm8RQ"
"7hFYs6y-QG6wGYEkoBKmdg"
"aELtGN6DQpmdRlQbr8i0uA"
"Fx4l6_axSGCxpyFm7C7BSQ"
"HAFmGcWuQAylxGjmnZZkSQ"
"H-eP-33FREOtq7v0uBPWbQ"
"QRmY8R2MQemuePb0EkYxWA"
"wKy-YzB-TK2lnK86Sx2RBA"
"y2ZmJ-_GRAmi3eHy1y8jzw"
And here is es3.r7 with preference=_primary_first:
$curl -XGET
"http://esq3.r7/events/_search?q=sessionId:1390953880&size=100&preference=_primary_first"
| jq '.hits.hits._id' | sort
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 19335 100 19335 0 0 871k 0 --:--:-- --:--:-- --:--:--
899k
"0LcI_px4SZy5ZQkI_V7Qyw"
"1sAGREtMSfK8OIxZErm8RQ"
"6IV2v4TFTr-Gl1eC6hrj0Q"
"6nwMexTHQBmFxfykOgKqWA"
"7hFYs6y-QG6wGYEkoBKmdg"
"9MTM10SeQ2yqWIb08oPnFA"
"aELtGN6DQpmdRlQbr8i0uA"
"AUHUg6k0QZOf_oGjsjSsGA"
"Bo_u1eYGSF2LeU78kbcFZg"
"EWs1K8YsR9-IBSAWK6ld7A"
"Fx4l6_axSGCxpyFm7C7BSQ"
"gpCrAZrNTNezWPfensER3g"
"HAFmGcWuQAylxGjmnZZkSQ"
"HB4Kwz3RSWWH5NHvyH4JMg"
"H-eP-33FREOtq7v0uBPWbQ"
"_IH6W4DoTRmdms0FJNlg4g"
"iK_3TbzcSj2-MbMXip_XFg"
"J4bjPFIcQ1ewrQqjN2qz6Q"
"kfonMDBuR--UIhkyM2cWrg"
"Kr6-9-3uR9Wp2923n-O2NA"
"Nw_9rjwvQ62u-HsuWIm53A"
"QRmY8R2MQemuePb0EkYxWA"
"usloSJzQRzCpOQ8bxKi2vA"
"w9NGEWg-QiivMpjyurYKrA"
"wKy-YzB-TK2lnK86Sx2RBA"
"y2ZmJ-_GRAmi3eHy1y8jzw"
"ZmFj7w4hR5Cvy-owCLmZ1Q"
"ZmlndPBLT-ivuOxm_A7yDA"
On Thursday, January 30, 2014 4:00:49 PM UTC-8, tallpsmith wrote:
If you can narrow down a specific few IDs of results that appear/disappear
based on the primary/replica shard, and confirm through an explicit GET of
that ID with the preference=_local on the primary shard & replica for that
result. To work out which shard # a specific ID belongs to, you can run
this query:
curl -XGET 'http://127.0.0.1:9200/_all/_search?pretty=1
http://127.0.0.1:9200/_all/_search?pretty=1' -d '
{
"fields" : ,
"query" : {
"ids" : {
"values" : [
"123456789"
]
}
},
"explain" : 1
}
'
where the "values" attribute you place the ID of the item you're after.
Within the result response you'l see the shard Id, use that to identify
which host is the primary and which is the replica. You can then run the
GET query with the preference=_local on each of those hosts and see if the
primary or replica shows the result. You will need to understand whether
the item that is 'flappy' (appearing/disappearing depending on the shard
being searched) is supposed to be in there or not, perhaps checking the
data store that is the source of the index (is it a dB?).
We have very infrequent case where the replica shard is not properly
receiving a delete at least with 0.19.10. The delete successfully applies
to the Primary, but the Replica still holds the value and returns it within
search results. We have loads of insert/update/delete activity and the
number of flappy items is very small, but it is definitely a thing.
see this previous thread:
http://elasticsearch-users.115913.n3.nabble.com/Deleted-items-appears-when-searching-a-replica-shard-td4029075.html
If it is the replica shard that's incorrect (my bet), the way to fix it is
to relocate the replica shard to another host. The relocation will take
the copy of the primary (correct copy) and recreate a new replica shard,
effectively neutralizing the inconsistency.
We have written a tool, Scrutineer (https://github.com/aconex/scrutineer)
which can help detect inconsistencies in your cluster. I also have a tool
not yet published to github that can help check these Primary/Replica
inconsistencies if that would help (you pass a list of IDs to it and it'll
check whether they're flappy between the primary & replica or not). It can
also help automate the rebuilding of just the replica shards by shunting
them around (rather than a full rolling restart of ALL the shards, just the
shard replicas you want)
cheers,
Paul Smith
On 31 January 2014 09:44, Binh Ly <bi...@hibalo.com <javascript:>> wrote:
Xavier, can you post an example of 1 full query and then also show how
the results of this one query is inconsistent? Just trying to understand
what is inconsistent. Thanks.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/038735ba-cef6-4634-9d46-7ff39dffc4d2%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a99ab249-ddf4-4c38-97d7-3bfe8ec41b5f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
