Inconsistent results while querying on a index

[RenuGuru1]

We are using elasticsearch 1.6.0 with 6 nodes.

We are getting inconsistent results while searching when the same query is requested concurrently, the order of results are changing.

Results are entirely different for search from 80 and size 2 and another search for the same query from 81 and size 2. While the 81st result should appear in both searches.

This doesn't occur when search is made only on primary shards. Replicas become inconsistent in a while.

We faced this same issue from the versions 0.90.x as well, This issue is on many versions. We are facing the same issue again on 1.6.0. On earlier versions, we solved it by making the index replica to 0 and then increasing to desired number. But since this is very risky to do it on indices of production environment, please suggest any other alternative solution to fix this inconsistency ASAP, without much risk on production cluster.

Also, Can you please fix these inconsistency issues in future release?

Thanks.

[jpountz (Adrien Grand)]

What sort order are you using? When the sort value is the same on two documents, then the order in which documents are returned might different based on the queried shard. In order to make this issue less visible, you can make each user always use the same preference so that the same shard copy will be queried. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-preference.html

[RenuGuru1]

@jpountz, Thanks for the reply.

"What sort order are you using? " - I experienced this without using any kind of sorting.

"you can make each user always use the same preference so that the same shard copy will be queried." - I was thinking of using _primary_first preference for all calls, does it have any impact on performance when compared to calling without preference?

[jpountz (Adrien Grand)]

@RenuGuru1

"you can make each user always use the same preference so that the same shard copy will be queried." - I was thinking of using primaryfirst preference for all calls, does it have any impact on performance when compared to calling without preference?

Using primary_first for all calls makes little sense if you have replicas as it will prevent elasticsearch from using replicas to scale reads. It would be better to use eg. the user id as a preference for each search request.

[RenuGuru1]

Yes! this makes sense, a question on how userId/sessionId as preference works. Say I'm using sessionId as preference, when the user searches, it hits some shards and gets results. on another search the same shards will be used for getting results for this user.

During this time, if a new document is indexed and stored in another shard different from the user's pre-allocated shards. This newly indexed documents will not be returned on search right? Yeah, this will not cause much issue as the sessionId differs and will be searched on different shards, each time a user logins'. But would like to know how this works, please enlighten me. Thanks.

[jpountz (Adrien Grand)]

I think you are confusing preference and routing.

Say you have an index with 2 primary shards, that have one replica shard each. This is a total of 4 shards. Let's call the primary shards 0P and 1P, and the replica shards 0R and 1R.

Routing by user id would allow you to make sure that all documents from the same user id will be on the same shard. For instance maybe all documents from user foo would be on shards 0P and 0R while all documents from user bar would end up on shards 1P and 1R. So at search time, if you know which user you are searching on, you can configure routing and only go to the shard that holds data for this user.

On the contrary, preference doesn't do anything at index time. But at search time it will make sure that you always hit the same shards if you specify the same preference. Without preference the first search request might go to shards 0P and 1R while the second search request will go to 0R and 1R. With preference configured, you are guaranteed to always hit the same shards.

[RenuGuru1]

@Andrien, Yes I understand routing and preference are different.
I think it would be better to understand if I give you an example of how we use elasticsearch, lets say we use it for shopping cart where user searches for all of our contents/products in our website search page. The sequence of the products displayed gets changed on concurrent requests. This is the kind of issue we are facing.

Considering this case, I'm not talking about routing since we are not storing user specific contents. We want some users to search for all of our products.

Now,

As you said, say I have an index with 2 primary shards, that have one replica shard each. This is a total of 4 shards. Let's call the primary shards 0P and 1P, and the replica shards 0R and 1R.

Now, with user's sessionId as preference, on his first search if shards 0R and 1P are searched, and
parallely if on indexing, new products gets indexed on 0P and 1P. Before it gets replicated, if the user does a second search, would the products indexed on 0P be returned?

[RenuGuru1]

And it is obvious that some non sync has happened while elasticsearch replicates, why and when does this non sync of replicas happen? It would be very helpful if you can avoid this non sync issue from elasticsearch itself, so that every user can see the products in same sequence.

[jpountz (Adrien Grand)]

If you want every user to see products in the same sequence then you need to change your sort order to tie break on the _uid field, so that the sort order is deterministic even in case a different replica is queried or segments are being merged.

[RenuGuru1]

Adding our conversation here for reference,

@RenuGuru1

Hi Grand,

Let me understand. Say when user searches with some keyword, we will show products ranked based on lucene's text relevance of the keyword, usage of the product (views/timespent) and metadata weight like if a product doesn't have description it will be demoted compared to documents with description. All these kind of weightage buckets are considered while ranking. So, how does this _uid field sort order help? I don't get it.

@jpountz

It will help by forcing elasticsearch to return documents in a deterministic order if your documents have the same score, usage and metadata so that merges or querying a different replica won't have an impact on the order in which documents are returned. It would look something like

{
"sort": [
_score,
{ "views": "asc" },
{ "_uid": "asc" } // tie-break
]
}

@RenuGuru1

Nice! I can try this in future, when I face the same issue on documents with same score, but I see documents with completely different scores, so my guess is few documents are missing in replica while they are available in primary.

Lets say a document d1 is in 43th position in primary shards, while it is in 37th position in replica. We search for the doc d1 from 35 to 40 in first call, and 41 to 45 in second call. Now, if first call searches on replica and second call searches on primary shards, the doc d1 will appear on both result sets, this d1 is appearing twice as a duplicate while scrolling on the website.

@jpountz

Right, this is why using preference is useful in addition to tie-breaking on the _uid.

Differences in ranking may happen between the primary and the replica because they can refresh at different times and because they can have different deleted documents which are still taken into account for index statistics, but the differences should decrease as your index grows.